Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.

@borisbaldassari I've got a new issue in the docker run now [1]

It looks like the listing is sending datetime. It should send iso8601 string imsmw (it's the loader which is in charge of converting the iso string into a datetime).

[1] P1280

@borisbaldassari I've got a new issue in the docker run now [1]

In T3746#78092, @ardumont wrote:

I'll have to find or craft a better fld file for the docker environment, with valid
poms that provide valid git repos. Will do asap. Please note however that we could use
one of the many fld files provided at [1], but I'd rather validate the whole run with
a simplistic subset.

Agreed about the simplistic subset, thanks in advance.

Note: I missed your [1] reference ¯\_(ツ)_/¯

Something's still fishy, since I took care of re-installing my venv with the latest
requirements.txt. But now with an updated version of mypy it works like a charm.
Thanks.

Update: I could run docker-dev, and there are indeed two maven entries found.
Unfortunately the only poms available in the example fld file are invalid and as a result the parser doesn't find any git/scm entry. I'll have to find or craft a better fld file for the docker environment, with valid poms that provide valid git repos. Will do asap.

In T3746#77998, @ardumont wrote:

I'm now convinced your blocking point is about ^. I reproduced it with mypy<0.920 [4]

Wow. Brilliant. @ardumont, you're great! :-)

@borisbaldassari hello, I'll first answer generally on what you said and did, awesome!

Anyway, the only code I see is [3] and I guess from that code we dont get through the
conditional somehow?

[3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/maven/lister.py$228-231

In T3746#77808, @ardumont wrote:

yes, that'd be a case for just adding that new kind of input in the lister test code and
then fix it in a diff.

I've done that (fixed case, added test), but can't commit:

@borisbaldassari in case you did not notice earlier ^ (i keep on forgetting to highlight you ;)

• a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

Ok, got it: unluckily this one string is viciously malformed (it's wrong, but fits the
regexp anyway), and I had not considered this very case. Looking at the bright side,
this shows a tricky bug that will be fixed.

In T3746#77610, @ardumont wrote:

• missing last_update entries in the task (I recalled we discussed and fixes those in the related diffs)

In T3746#77695, @borisbaldassari wrote:

Still, there is something fishy about:

• a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

That sounds ok to me: scm entries are deduplicated. 2 pages corresponding to the same scm entry (i.e. with the same scm url) will produce only one output.
In this sample we should indeed have 2 jars and 1 scm.

The fix works on my setup, thanks again. :-)

I've opened D7025 to improve some logging instructions and add what was missing.

Thanks a lot for the advances made @ardumont

Thanks a lot for the advances made @ardumont
If I understand correctly, we're missing some jar entries in the exported maven repositories. I'll figure that out and add some tonight.

After D7023, the scheduling happens:

Connecting to the rabbitmq admin page [1], we can see the messages in the ready state (so not consumed indeed).

• Update the docker stack to the current repositories' heads

$ swh-doco-rebuild
+ DOCKER_CMD=/nix/store/r50lif10qfix73c3rxvjbmsd1x8v45rj-docker-20.10.12/bin/docker
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ /nix/store/r50lif10qfix73c3rxvjbmsd1x8v45rj-docker-20.10.12/bin/docker build -f Dockerfile --no-cache -t swh/stack .
Sending build context to Docker daemon     99MB
Step 1/13 : FROM python:3.7
 ---> ad37de9b03ef
...

• Check the scheduler state after docker up:

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

@borisbaldassari Hello, gentle ping about ^

@stsp as mentioned in the main task, it'd be neat if you could create yourself an account in sentry [1].
Ping me when it's done so i can invite you in the #swh-team so you can browse the cvs loader
related issues from there.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual
machine alongside the swh setup to host the text index file, and make sure the swh vm
could access it. I suppose that any vm/docker/baremetal machine with an apache/nginx
server could do for that, as long as the lister can http-fetch the .fld file.

This PDF contains my SVN Loader review report.

This has been done though the debian improvment tooling task [1].
Jenkins builds and uploads debian package as usual after a git tag on the repository.

I found one additional problem. See D6823.

Unless I have overlooked something, all currently known issues have now been addressed.

Unless I have overlooked something, all currently known issues have now been addressed.