Yesss! \o/

Good news *can* happen, ahah! Thanks for notifying me.

I'm not sure to understand the intent, as we already keep the pom in the extrinsic metadata (don't we?).
Double-checking in the SWH codebase, I believe you could build upon this: see [1] lines 166-180.

Congrats on the work done! I think that downloading the pom file from the same folder is indeed the way to go.

I see there is a lot of progress here, nice!
I try to follow the thread as time allows, but if you're stuck please do not hesitate to notify me.

Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.

In D7052#184773, @ardumont wrote:

ImportError: cannot import name 'TempdirFactory' from '_pytest.tmpdir'

unrelated to your current diff. See D7084.

Ok, thanks. This is always good to hear, ahah.

• maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
• maven: Fix undef last_update in ListedOrigins.

In D7052#184250, @ardumont wrote:

To be clear, it's ok to land now.

Sorry for the latence, and thanks for clarifying.

Try to re-trigger the build.

Update commits

• maven: Fix undef last_update in ListedOrigins.
• maven: Dismiss origins if they are malformed - e.g. wrong pom scm format, add test.

In D7052#183629, @ardumont wrote:

Create as much commits you need in the same branch.
Then you just need to be consistent with your arc diff use.

Oh, ok. Neat. Thanks for advising.
Actually there will only be 2 diffs: one for last_udpate, the other for the scm fix. I had rebased the diff in-between to merge the two commits related to last_update, but failed to update my comments.

• maven: dismiss origins if they are malformed - e.g. wrong pom scm format, add test.
• maven: fix undef last_update in ListedOrigins.

In D7052#183474, @ardumont wrote:

I'm confused by one of the test code change. And I have some other suggestions inline ;)

Explained and fixed the comments.

In T3746#78092, @ardumont wrote:

I'll have to find or craft a better fld file for the docker environment, with valid
poms that provide valid git repos. Will do asap. Please note however that we could use
one of the many fld files provided at [1], but I'd rather validate the whole run with
a simplistic subset.

Agreed about the simplistic subset, thanks in advance.

Note: I missed your [1] reference ¯\_(ツ)_/¯

Update: I could run docker-dev, and there are indeed two maven entries found.
Unfortunately the only poms available in the example fld file are invalid and as a result the parser doesn't find any git/scm entry. I'll have to find or craft a better fld file for the docker environment, with valid poms that provide valid git repos. Will do asap.

• Fix last_update tzinfo.

In T3746#77998, @ardumont wrote:

I'm now convinced your blocking point is about ^. I reproduced it with mypy<0.920 [4]

Wow. Brilliant. @ardumont, you're great! :-)

Anyway, the only code I see is [3] and I guess from that code we dont get through the
conditional somehow?

[3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/maven/lister.py$228-231

In T3746#77808, @ardumont wrote:

yes, that'd be a case for just adding that new kind of input in the lister test code and
then fix it in a diff.

I've done that (fixed case, added test), but can't commit:

In T3746#77610, @ardumont wrote:

• missing last_update entries in the task (I recalled we discussed and fixes those in the related diffs)

In T3746#77695, @borisbaldassari wrote:

Still, there is something fishy about:

• a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

The fix works on my setup, thanks again. :-)

Thanks a lot for the advances made @ardumont
If I understand correctly, we're missing some jar entries in the exported maven repositories. I'll figure that out and add some tonight.

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

Yes, I had a conflict with the master's updates which messed up everything -- well, I messed up, to be precise.
It should be ok now, and the diff has been landed on master.

Update diff + rebase

rebase and fix conflicts from master

• swh-search: upgrade elasticsearch to 7.15.2
• docker/conf/loader: Add missing LoadNixguix task
• stop to use the deprecated docker-compose --no-ansi command line option
• Add durations to nginx and swh-storage access log
• Ensure swh-web is started before trying to refresh the save code now statuses
• Increase the swh-web timeout for swh-storage requests
• add libcmph-dev so that swh-perfecthash can build and install
• doc: transform markdown links into rst links
• codespell: cmak is not a spelling mistake
• Overhaul debian packaging bootstrap scripts
• Merge master into T1724_docker_dev

• Merge branch 'master' of https://forge.softwareheritage.org/source/swh-environment into T1724_docker_dev

merge from master

Merge from master

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual machine alongside the swh setup to host the text index file, and make sure the swh vm could access it.
I suppose that any vm/docker/baremetal machine with an apache/nginx server could do for that, as long as the lister can http-fetch the .fld file.

In T1724#75125, @ardumont wrote:

I'm asking you for a diff with the exact changes you had to make in the
swh-environment/docker/docker-compose.yml (and other folders) to actually make it run.
That will definitely help for the deployment on staging.

Makes sense. LGTM.
Thanks @vlorentz for the improvement.
(oh, I didn't click on submit, sorry)

Definitely agree that artefacts deleted on the server should return an empty snapshot.

As said in the lister's corresponding diff, makes sense to me. Thanks for the improvement.

LGTM. Considering the refactoring and removal of the extid, you fixed the proper maven parts in my humble opinion.

I'm not sure what you mean by the docker diff. Is that the update of the maven-index-exporter repository at D6740?
The above-mentioned repository has documentation to build, test and run the text index generation. As mentioned there I've also created a bunch of compressed text index exports, that can be used to test the lister/loader without running the docker image immediately. They are all real-world extracts obtained by running the docker image on the list of Maven repositories I could get as of last week. They together represent a few million artefacts.

In D6740#175399, @vlorentz wrote:

Could you use the original commits instead of squashing them?

• Add third-party license file for Apache License v2 requirements.
• Minor fixes in examples and docker script
• Add index update in test data with deleted artefacts.
• Add index update in test data with deleted artefacts.
• Refactor: move tests into tests/ directory
• Refactor: move resources to scripts
• Refactor: add docs
• Update list of repositories, fix doc typo
• Update doc
• Update doc
• Update doc

In D6396#175519, @vlorentz wrote:

You need to update your diff after rebasing

Update before push to master

In D6396#175212, @ardumont wrote:

@borisbaldassari Can you please push your commit in master?

Fix unordered function return + rebase

• maven-loader: fix tests for unordered function call

• loader: add new maven-jar loader
• maven-loader: set the pom as metadata
• maven-loader: fix after final review (ardumont, vlorentz)
• maven-loader: fix metadata for completeness
• maven-loader: fix metadata for pom and json
• maven-loader: move request.get to own function
• maven-loader: fix tests for unordered function call

• maven-loader: move request.get to own function

Rebase with latest changes from master

• loader: add new maven-jar loader
• maven-loader: set the pom as metadata
• maven-loader: fix after final review (ardumont, vlorentz)
• maven-loader: fix metadata for completeness
• maven-loader: fix metadata for pom and json

• maven-loader: fix metadata for pom and json

Updated encode to fix double metadata objects.

Rebase.

• maven-loader: fix metadata for completeness

• maven-loader: set the pom as metadata
• maven-loader: fix after final review (ardumont, vlorentz)