Page MenuHomeSoftware Heritage
Feed Advanced Search

Jun 15 2022

borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Yesss! \o/

Jun 15 2022, 4:33 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Good news *can* happen, ahah! Thanks for notifying me.

Jun 15 2022, 3:42 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

I'm not sure to understand the intent, as we already keep the pom in the extrinsic metadata (don't we?).
Double-checking in the SWH codebase, I believe you could build upon this: see [1] lines 166-180.

Jun 15 2022, 3:07 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T4326: Archive the pom file additionally to the source folder.

Congrats on the work done! I think that downloading the pom file from the same folder is indeed the way to go.

Jun 15 2022, 2:54 PM · Maven loader, Maven lister, System administration, Archive coverage

Apr 1 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I see there is a lot of progress here, nice!
I try to follow the thread as time allows, but if you're stuck please do not hesitate to notify me.

Apr 1 2022, 6:10 PM · Maven loader, Maven lister, System administration, Archive coverage

Feb 13 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.

Feb 13 2022, 11:09 AM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari requested review of D7167: maven-docker: update export file to include svn/git entries..
Feb 13 2022, 10:56 AM
borisbaldassari added a revision to T1724: Maven Central repository support: D7167: maven-docker: update export file to include svn/git entries..
Feb 13 2022, 10:56 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Feb 8 2022

borisbaldassari added a comment to D7052: Maven: fix lister after docker-dev review..
ImportError: cannot import name 'TempdirFactory' from '_pytest.tmpdir'

unrelated to your current diff. See D7084.

Ok, thanks. This is always good to hear, ahah.

Feb 8 2022, 8:01 AM
borisbaldassari committed rDLSd4e1e8212ace: maven: Fix undef last_update in ListedOrigins. (authored by borisbaldassari).
maven: Fix undef last_update in ListedOrigins.
Feb 8 2022, 7:58 AM
borisbaldassari closed D7052: Maven: fix lister after docker-dev review..
Feb 8 2022, 7:58 AM
borisbaldassari committed rDLS24eeabfadec2: maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add… (authored by borisbaldassari).
maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add…
Feb 8 2022, 7:58 AM
borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..
  • maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
  • maven: Fix undef last_update in ListedOrigins.
Feb 8 2022, 7:53 AM

Feb 4 2022

borisbaldassari added a comment to D7052: Maven: fix lister after docker-dev review..

To be clear, it's ok to land now.

Sorry for the latence, and thanks for clarifying.

Feb 4 2022, 10:33 PM
borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..

Try to re-trigger the build.

Feb 4 2022, 10:32 PM
borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..

Update commits

Feb 4 2022, 10:23 PM
borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..
  • maven: Fix undef last_update in ListedOrigins.
  • maven: Dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
Feb 4 2022, 10:17 PM

Feb 3 2022

borisbaldassari added a comment to D7052: Maven: fix lister after docker-dev review..

Create as much commits you need in the same branch.
Then you just need to be consistent with your arc diff use.

Oh, ok. Neat. Thanks for advising.
Actually there will only be 2 diffs: one for last_udpate, the other for the scm fix. I had rebased the diff in-between to merge the two commits related to last_update, but failed to update my comments.

Feb 3 2022, 8:45 AM

Feb 2 2022

borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..
  • maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
  • maven: fix undef last_update in ListedOrigins.
Feb 2 2022, 8:33 AM
borisbaldassari updated the summary of D7052: Maven: fix lister after docker-dev review..
Feb 2 2022, 8:25 AM
borisbaldassari added a comment to D7052: Maven: fix lister after docker-dev review..

I'm confused by one of the test code change. And I have some other suggestions inline ;)

Explained and fixed the comments.

Feb 2 2022, 8:19 AM

Feb 1 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I'll have to find or craft a better fld file for the docker environment, with valid
poms that provide valid git repos. Will do asap. Please note however that we could use
one of the many fld files provided at [1], but I'd rather validate the whole run with
a simplistic subset.

Agreed about the simplistic subset, thanks in advance.

Note: I missed your [1] reference ¯\_(ツ)_/¯

Feb 1 2022, 10:03 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 31 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Update: I could run docker-dev, and there are indeed two maven entries found.
Unfortunately the only poms available in the example fld file are invalid and as a result the parser doesn't find any git/scm entry. I'll have to find or craft a better fld file for the docker environment, with valid poms that provide valid git repos. Will do asap.

Jan 31 2022, 10:27 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari updated the diff for D7052: Maven: fix lister after docker-dev review..
  • Fix last_update tzinfo.
Jan 31 2022, 10:18 PM
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I'm now convinced your blocking point is about ^. I reproduced it with mypy<0.920 [4]

Wow. Brilliant. @ardumont, you're great! :-)

Jan 31 2022, 9:42 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari requested review of D7052: Maven: fix lister after docker-dev review..
Jan 31 2022, 8:47 PM
borisbaldassari added a revision to T3746: staging: Deploy maven indexer/lister/loader: D7052: Maven: fix lister after docker-dev review..
Jan 31 2022, 8:44 PM · Maven loader, Maven lister, System administration, Archive coverage

Jan 30 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Anyway, the only code I see is [3] and I guess from that code we dont get through the
conditional somehow?

[3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/maven/lister.py$228-231

Jan 30 2022, 10:25 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

yes, that'd be a case for just adding that new kind of input in the lister test code and
then fix it in a diff.

I've done that (fixed case, added test), but can't commit:

Jan 30 2022, 1:47 PM · Maven loader, Maven lister, System administration, Archive coverage

Jan 26 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.
  • missing last_update entries in the task (I recalled we discussed and fixes those in the related diffs)
Jan 26 2022, 12:04 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 25 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Still, there is something fishy about:

  • a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

Jan 25 2022, 10:40 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

The fix works on my setup, thanks again. :-)

Jan 25 2022, 9:32 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 24 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Thanks a lot for the advances made @ardumont
If I understand correctly, we're missing some jar entries in the exported maven repositories. I'll figure that out and add some tonight.

Jan 24 2022, 12:41 PM · Maven loader, Maven lister, System administration, Archive coverage

Jan 21 2022

borisbaldassari updated the task description for T1724: Maven Central repository support.
Jan 21 2022, 4:09 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari closed D7014: Fix comment in doco.yml.
Jan 21 2022, 4:08 PM
borisbaldassari committed rDENVc160ade2e2a4: Fix comment in doco.yml (authored by borisbaldassari).
Fix comment in doco.yml
Jan 21 2022, 4:08 PM
borisbaldassari requested review of D7014: Fix comment in doco.yml.
Jan 21 2022, 4:03 PM
borisbaldassari added a revision to T1724: Maven Central repository support: D7014: Fix comment in doco.yml.
Jan 21 2022, 4:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari closed D6955: maven: add index nginx host in docker dev setup.
Jan 21 2022, 3:55 PM
borisbaldassari committed rDENV160465ffc5f4: maven: add index host in docker dev setup (authored by borisbaldassari).
maven: add index host in docker dev setup
Jan 21 2022, 3:55 PM

Jan 15 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Jan 15 2022, 8:21 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari requested review of D6955: maven: add index nginx host in docker dev setup.
Jan 15 2022, 8:06 PM
borisbaldassari added a revision to T1724: Maven Central repository support: D6955: maven: add index nginx host in docker dev setup.
Jan 15 2022, 8:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 10 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

Jan 10 2022, 8:39 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 16 2021

borisbaldassari added a comment to D6784: maven: diff docker dev setup.

Yes, I had a conflict with the master's updates which messed up everything -- well, I messed up, to be precise.
It should be ok now, and the diff has been landed on master.

Dec 16 2021, 8:28 AM
borisbaldassari closed D6784: maven: diff docker dev setup.
Dec 16 2021, 8:27 AM
borisbaldassari committed rDENV3dfeb0036c00: maven: diff docker dev setup (authored by borisbaldassari).
maven: diff docker dev setup
Dec 16 2021, 8:27 AM
borisbaldassari updated the diff for D6784: maven: diff docker dev setup.

Update diff + rebase

Dec 16 2021, 8:26 AM
borisbaldassari updated the diff for D6784: maven: diff docker dev setup.

rebase and fix conflicts from master

Dec 16 2021, 8:19 AM

Dec 15 2021

borisbaldassari updated the diff for D6784: maven: diff docker dev setup.
  • swh-search: upgrade elasticsearch to 7.15.2
  • docker/conf/loader: Add missing LoadNixguix task
  • stop to use the deprecated docker-compose --no-ansi command line option
  • Add durations to nginx and swh-storage access log
  • Ensure swh-web is started before trying to refresh the save code now statuses
  • Increase the swh-web timeout for swh-storage requests
  • add libcmph-dev so that swh-perfecthash can build and install
  • doc: transform markdown links into rst links
  • codespell: cmak is not a spelling mistake
  • Overhaul debian packaging bootstrap scripts
  • Merge master into T1724_docker_dev
Dec 15 2021, 7:05 PM
borisbaldassari updated the diff for D6784: maven: diff docker dev setup.
Dec 15 2021, 6:51 PM
borisbaldassari updated the diff for D6784: maven: diff docker dev setup.

merge from master

Dec 15 2021, 6:48 PM
borisbaldassari updated the diff for D6784: maven: diff docker dev setup.

Merge from master

Dec 15 2021, 6:43 PM

Dec 8 2021

borisbaldassari added a comment to T1724: Maven Central repository support.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual machine alongside the swh setup to host the text index file, and make sure the swh vm could access it.
I suppose that any vm/docker/baremetal machine with an apache/nginx server could do for that, as long as the lister can http-fetch the .fld file.

Dec 8 2021, 12:18 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a comment to T1724: Maven Central repository support.

I'm asking you for a diff with the exact changes you had to make in the
swh-environment/docker/docker-compose.yml (and other folders) to actually make it run.
That will definitely help for the deployment on staging.

Dec 8 2021, 12:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari requested review of D6784: maven: diff docker dev setup.
Dec 8 2021, 11:58 AM
borisbaldassari added a revision to T1724: Maven Central repository support: D6784: maven: diff docker dev setup.
Dec 8 2021, 11:58 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari accepted D6770: maven: Pass the base URL of the Maven instance to the loader.

Makes sense. LGTM.
Thanks @vlorentz for the improvement.
(oh, I didn't click on submit, sorry)

Dec 8 2021, 10:31 AM

Dec 7 2021

borisbaldassari accepted D6769: maven: Don't carry deleted versions over to the next snapshot.

Definitely agree that artefacts deleted on the server should return an empty snapshot.

Dec 7 2021, 5:08 PM
borisbaldassari committed rDLSMAVEXP8706d2ea6a7c: Update doc (authored by borisbaldassari).
Update doc
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP5fbccbf04f1c: Update doc (authored by borisbaldassari).
Update doc
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP22be65e78173: Update doc (authored by borisbaldassari).
Update doc
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXPc8f1af6ba879: Update list of repositories, fix doc typo (authored by borisbaldassari).
Update list of repositories, fix doc typo
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXPe1469f485c88: Refactor: add docs (authored by borisbaldassari).
Refactor: add docs
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXPad4e81d67d0a: Refactor: move resources to scripts (authored by borisbaldassari).
Refactor: move resources to scripts
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP71a80b9cb1d7: Refactor: move tests into tests/ directory (authored by borisbaldassari).
Refactor: move tests into tests/ directory
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP33c4f614b318: Add index update in test data with deleted artefacts. (authored by borisbaldassari).
Add index update in test data with deleted artefacts.
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP235f3b290498: Add index update in test data with deleted artefacts. (authored by borisbaldassari).
Add index update in test data with deleted artefacts.
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP873abda79c3c: Minor fixes in examples and docker script (authored by borisbaldassari).
Minor fixes in examples and docker script
Dec 7 2021, 4:49 PM
borisbaldassari closed D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
Dec 7 2021, 4:49 PM
borisbaldassari committed rDLSMAVEXP02e1e9c63dc9: Add third-party license file for Apache License v2 requirements. (authored by borisbaldassari).
Add third-party license file for Apache License v2 requirements.
Dec 7 2021, 4:49 PM
borisbaldassari added a comment to D6771: maven: Use the instance base_url as metadata authority URL.

As said in the lister's corresponding diff, makes sense to me. Thanks for the improvement.

Dec 7 2021, 4:23 PM
borisbaldassari accepted D6761: maven: Remove dead code for extid computation.

LGTM. Considering the refactoring and removal of the extid, you fixed the proper maven parts in my humble opinion.

Dec 7 2021, 12:09 PM
borisbaldassari added a comment to D6761: maven: Remove dead code for extid computation.
Dec 7 2021, 12:08 PM

Dec 6 2021

borisbaldassari added a comment to T1724: Maven Central repository support.

I'm not sure what you mean by the docker diff. Is that the update of the maven-index-exporter repository at D6740?
The above-mentioned repository has documentation to build, test and run the text index generation. As mentioned there I've also created a bunch of compressed text index exports, that can be used to test the lister/loader without running the docker image immediately. They are all real-world extracts obtained by running the docker image on the list of Maven repositories I could get as of last week. They together represent a few million artefacts.

Dec 6 2021, 10:16 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari updated the summary of D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
Dec 6 2021, 5:48 PM
borisbaldassari added a comment to D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.

Could you use the original commits instead of squashing them?

Dec 6 2021, 5:32 PM
borisbaldassari updated the diff for D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
  • Add third-party license file for Apache License v2 requirements.
  • Minor fixes in examples and docker script
  • Add index update in test data with deleted artefacts.
  • Add index update in test data with deleted artefacts.
  • Refactor: move tests into tests/ directory
  • Refactor: move resources to scripts
  • Refactor: add docs
  • Update list of repositories, fix doc typo
  • Update doc
  • Update doc
  • Update doc
Dec 6 2021, 5:31 PM
borisbaldassari added a comment to D6396: Implement maven jar source files loader.

You need to update your diff after rebasing

Dec 6 2021, 2:47 PM
borisbaldassari closed D6396: Implement maven jar source files loader.
Dec 6 2021, 2:46 PM
borisbaldassari committed rDLDBASE89f5ccc7f5fc: loader: add new maven-jar loader (authored by borisbaldassari).
loader: add new maven-jar loader
Dec 6 2021, 2:45 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

Update before push to master

Dec 6 2021, 2:45 PM
borisbaldassari added a comment to D6396: Implement maven jar source files loader.

@borisbaldassari Can you please push your commit in master?

Dec 6 2021, 12:21 PM

Dec 3 2021

borisbaldassari updated the summary of D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
Dec 3 2021, 1:58 PM
borisbaldassari requested review of D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
Dec 3 2021, 1:57 PM
borisbaldassari added a revision to T1724: Maven Central repository support: D6740: Update maven-index-exporter from gh/borisbaldassari/maven-index-exporter.
Dec 3 2021, 1:57 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 1 2021

borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

Fix unordered function return + rebase

Dec 1 2021, 1:54 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • maven-loader: fix tests for unordered function call
Dec 1 2021, 1:42 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • loader: add new maven-jar loader
  • maven-loader: set the pom as metadata
  • maven-loader: fix after final review (ardumont, vlorentz)
  • maven-loader: fix metadata for completeness
  • maven-loader: fix metadata for pom and json
  • maven-loader: move request.get to own function
  • maven-loader: fix tests for unordered function call
Dec 1 2021, 1:37 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • maven-loader: move request.get to own function
Dec 1 2021, 12:57 PM
borisbaldassari updated the task description for T1724: Maven Central repository support.
Dec 1 2021, 10:35 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Nov 29 2021

borisbaldassari closed D6395: lister: Add new maven lister.
Nov 29 2021, 5:40 PM
borisbaldassari committed rDLS8991c625ea56: lister: Add new maven lister (authored by borisbaldassari).
lister: Add new maven lister
Nov 29 2021, 5:40 PM
borisbaldassari updated the diff for D6395: lister: Add new maven lister.

Rebase with latest changes from master

Nov 29 2021, 5:35 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • loader: add new maven-jar loader
  • maven-loader: set the pom as metadata
  • maven-loader: fix after final review (ardumont, vlorentz)
  • maven-loader: fix metadata for completeness
  • maven-loader: fix metadata for pom and json
Nov 29 2021, 4:28 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • maven-loader: fix metadata for pom and json
Nov 29 2021, 4:06 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

Updated encode to fix double metadata objects.

Nov 29 2021, 3:40 PM
borisbaldassari updated the diff for D6395: lister: Add new maven lister.

Rebase.

Nov 29 2021, 3:31 PM
borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • maven-loader: fix metadata for completeness
Nov 29 2021, 3:18 PM

Nov 28 2021

borisbaldassari updated the diff for D6396: Implement maven jar source files loader.
  • maven-loader: set the pom as metadata
  • maven-loader: fix after final review (ardumont, vlorentz)
Nov 28 2021, 12:29 PM