User Details
- User Since
- Mar 5 2021, 12:08 PM (182 w, 1 d)
Jun 15 2022
Yesss! \o/
Good news *can* happen, ahah! Thanks for notifying me.
I'm not sure to understand the intent, as we already keep the pom in the extrinsic metadata (don't we?).
Double-checking in the SWH codebase, I believe you could build upon this: see [1] lines 166-180.
Congrats on the work done! I think that downloading the pom file from the same folder is indeed the way to go.
Apr 1 2022
I see there is a lot of progress here, nice!
I try to follow the thread as time allows, but if you're stuck please do not hesitate to notify me.
Feb 13 2022
Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.
Feb 8 2022
Ok, thanks. This is always good to hear, ahah.
- maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
- maven: Fix undef last_update in ListedOrigins.
Feb 4 2022
Sorry for the latence, and thanks for clarifying.
Try to re-trigger the build.
Update commits
- maven: Fix undef last_update in ListedOrigins.
- maven: Dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
Feb 3 2022
Oh, ok. Neat. Thanks for advising.
Actually there will only be 2 diffs: one for last_udpate, the other for the scm fix. I had rebased the diff in-between to merge the two commits related to last_update, but failed to update my comments.
Feb 2 2022
- maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
- maven: fix undef last_update in ListedOrigins.
Explained and fixed the comments.
Feb 1 2022
Jan 31 2022
Update: I could run docker-dev, and there are indeed two maven entries found.
Unfortunately the only poms available in the example fld file are invalid and as a result the parser doesn't find any git/scm entry. I'll have to find or craft a better fld file for the docker environment, with valid poms that provide valid git repos. Will do asap.
- Fix last_update tzinfo.
Wow. Brilliant. @ardumont, you're great! :-)
Jan 30 2022
Anyway, the only code I see is [3] and I guess from that code we dont get through the
conditional somehow?
I've done that (fixed case, added test), but can't commit:
Jan 26 2022
Jan 25 2022
The fix works on my setup, thanks again. :-)
Jan 24 2022
Thanks a lot for the advances made @ardumont
If I understand correctly, we're missing some jar entries in the exported maven repositories. I'll figure that out and add some tonight.
Jan 21 2022
Jan 15 2022
@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.
Jan 10 2022
Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!
Dec 16 2021
Yes, I had a conflict with the master's updates which messed up everything -- well, I messed up, to be precise.
It should be ok now, and the diff has been landed on master.
Update diff + rebase
rebase and fix conflicts from master
Dec 15 2021
- swh-search: upgrade elasticsearch to 7.15.2
- docker/conf/loader: Add missing LoadNixguix task
- stop to use the deprecated docker-compose --no-ansi command line option
- Add durations to nginx and swh-storage access log
- Ensure swh-web is started before trying to refresh the save code now statuses
- Increase the swh-web timeout for swh-storage requests
- add libcmph-dev so that swh-perfecthash can build and install
- doc: transform markdown links into rst links
- codespell: cmak is not a spelling mistake
- Overhaul debian packaging bootstrap scripts
- Merge master into T1724_docker_dev
- Merge branch 'master' of https://forge.softwareheritage.org/source/swh-environment into T1724_docker_dev
merge from master
Merge from master
Dec 8 2021
On second thoughts: in order to run the docker-dev setup, I also had to run a virtual machine alongside the swh setup to host the text index file, and make sure the swh vm could access it.
I suppose that any vm/docker/baremetal machine with an apache/nginx server could do for that, as long as the lister can http-fetch the .fld file.
Makes sense. LGTM.
Thanks @vlorentz for the improvement.
(oh, I didn't click on submit, sorry)
Dec 7 2021
Definitely agree that artefacts deleted on the server should return an empty snapshot.
As said in the lister's corresponding diff, makes sense to me. Thanks for the improvement.
LGTM. Considering the refactoring and removal of the extid, you fixed the proper maven parts in my humble opinion.
Dec 6 2021
I'm not sure what you mean by the docker diff. Is that the update of the maven-index-exporter repository at D6740?
The above-mentioned repository has documentation to build, test and run the text index generation. As mentioned there I've also created a bunch of compressed text index exports, that can be used to test the lister/loader without running the docker image immediately. They are all real-world extracts obtained by running the docker image on the list of Maven repositories I could get as of last week. They together represent a few million artefacts.
- Add third-party license file for Apache License v2 requirements.
- Minor fixes in examples and docker script
- Add index update in test data with deleted artefacts.
- Add index update in test data with deleted artefacts.
- Refactor: move tests into tests/ directory
- Refactor: move resources to scripts
- Refactor: add docs
- Update list of repositories, fix doc typo
- Update doc
- Update doc
- Update doc
Update before push to master
Dec 3 2021
Dec 1 2021
Fix unordered function return + rebase
- maven-loader: fix tests for unordered function call
- loader: add new maven-jar loader
- maven-loader: set the pom as metadata
- maven-loader: fix after final review (ardumont, vlorentz)
- maven-loader: fix metadata for completeness
- maven-loader: fix metadata for pom and json
- maven-loader: move request.get to own function
- maven-loader: fix tests for unordered function call
- maven-loader: move request.get to own function
Nov 29 2021
Rebase with latest changes from master
- loader: add new maven-jar loader
- maven-loader: set the pom as metadata
- maven-loader: fix after final review (ardumont, vlorentz)
- maven-loader: fix metadata for completeness
- maven-loader: fix metadata for pom and json
- maven-loader: fix metadata for pom and json
Updated encode to fix double metadata objects.
Rebase.
- maven-loader: fix metadata for completeness
Nov 28 2021
- maven-loader: set the pom as metadata
- maven-loader: fix after final review (ardumont, vlorentz)