Page MenuHomeSoftware Heritage
Feed Advanced Search

Feb 13 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Hi @ardumont, sorry for the delay, wild week here. And thanks for the iso 8601 fix.

Feb 13 2022, 11:09 AM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D7167: maven-docker: update export file to include svn/git entries..
Feb 13 2022, 10:56 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Feb 9 2022

ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

@borisbaldassari I've got a new issue in the docker run now [1]

It looks like the listing is sending datetime. It should send iso8601 string imsmw (it's the loader which is in charge of converting the iso string into a datetime).

[1] P1280

Feb 9 2022, 7:01 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 9 2022, 6:59 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 9 2022, 5:22 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a revision to T3746: staging: Deploy maven indexer/lister/loader: D7139: maven: Fix last update datetime.
Feb 9 2022, 5:01 PM · Maven loader, Maven lister, System administration, Archive coverage

Feb 8 2022

ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 8 2022, 2:03 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 8 2022, 2:03 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

@borisbaldassari I've got a new issue in the docker run now [1]

Feb 8 2022, 11:32 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 8 2022, 10:36 AM · Maven loader, Maven lister, System administration, Archive coverage

Feb 7 2022

ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Feb 7 2022, 5:59 PM · Maven loader, Maven lister, System administration, Archive coverage

Feb 1 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I'll have to find or craft a better fld file for the docker environment, with valid
poms that provide valid git repos. Will do asap. Please note however that we could use
one of the many fld files provided at [1], but I'd rather validate the whole run with
a simplistic subset.

Agreed about the simplistic subset, thanks in advance.

Note: I missed your [1] reference ¯\_(ツ)_/¯

Feb 1 2022, 10:03 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Something's still fishy, since I took care of re-installing my venv with the latest
requirements.txt. But now with an updated version of mypy it works like a charm.
Thanks.

Feb 1 2022, 9:54 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a revision to T3746: staging: Deploy maven indexer/lister/loader: D7054: developer-setup: Mention the current mypy version we use.
Feb 1 2022, 9:49 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 31 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Update: I could run docker-dev, and there are indeed two maven entries found.
Unfortunately the only poms available in the example fld file are invalid and as a result the parser doesn't find any git/scm entry. I'll have to find or craft a better fld file for the docker environment, with valid poms that provide valid git repos. Will do asap.

Jan 31 2022, 10:27 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I'm now convinced your blocking point is about ^. I reproduced it with mypy<0.920 [4]

Wow. Brilliant. @ardumont, you're great! :-)

Jan 31 2022, 9:42 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a revision to T3746: staging: Deploy maven indexer/lister/loader: D7052: Maven: fix lister after docker-dev review..
Jan 31 2022, 8:44 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

@borisbaldassari hello, I'll first answer generally on what you said and did, awesome!

Jan 31 2022, 9:15 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 30 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Anyway, the only code I see is [3] and I guess from that code we dont get through the
conditional somehow?

[3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/maven/lister.py$228-231

Jan 30 2022, 10:25 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

yes, that'd be a case for just adding that new kind of input in the lister test code and
then fix it in a diff.

I've done that (fixed case, added test), but can't commit:

Jan 30 2022, 1:47 PM · Maven loader, Maven lister, System administration, Archive coverage

Jan 28 2022

ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

@borisbaldassari in case you did not notice earlier ^ (i keep on forgetting to highlight you ;)

Jan 28 2022, 11:06 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 26 2022

ardumont updated subscribers of T3746: staging: Deploy maven indexer/lister/loader.
  • a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

Ok, got it: unluckily this one string is viciously malformed (it's wrong, but fits the
regexp anyway), and I had not considered this very case. Looking at the bright side,
this shows a tricky bug that will be fixed.

Jan 26 2022, 9:16 AM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.
  • missing last_update entries in the task (I recalled we discussed and fixes those in the related diffs)
Jan 26 2022, 12:04 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 25 2022

borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Still, there is something fishy about:

  • a visit_type 'https' (record 3 below [1]).

*that* is fishy. I'm investigating.

Jan 25 2022, 10:40 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont removed a project from T3874: staging: Analyze result of the maven listing and ingestion: System administration.
Jan 25 2022, 6:07 PM · Maven loader, Maven lister, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

That sounds ok to me: scm entries are deduplicated. 2 pages corresponding to the same scm entry (i.e. with the same scm url) will produce only one output.
In this sample we should indeed have 2 jars and 1 scm.

Jan 25 2022, 2:59 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

The fix works on my setup, thanks again. :-)

Jan 25 2022, 9:32 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 24 2022

ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

I've opened D7025 to improve some logging instructions and add what was missing.

Jan 24 2022, 2:44 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Thanks a lot for the advances made @ardumont

Jan 24 2022, 2:36 PM · Maven loader, Maven lister, System administration, Archive coverage
borisbaldassari added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Thanks a lot for the advances made @ardumont
If I understand correctly, we're missing some jar entries in the exported maven repositories. I'll figure that out and add some tonight.

Jan 24 2022, 12:41 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Jan 24 2022, 12:23 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Jan 24 2022, 12:19 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Jan 24 2022, 12:18 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

After D7023, the scheduling happens:

Jan 24 2022, 12:00 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a revision to T3746: staging: Deploy maven indexer/lister/loader: D7023: Add missing maven lister task subscription to lister config.
Jan 24 2022, 11:58 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.

Connecting to the rabbitmq admin page [1], we can see the messages in the ready state (so not consumed indeed).

Jan 24 2022, 11:43 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Jan 24 2022, 11:34 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T3746: staging: Deploy maven indexer/lister/loader.
  • Update the docker stack to the current repositories' heads
$ swh-doco-rebuild
+ DOCKER_CMD=/nix/store/r50lif10qfix73c3rxvjbmsd1x8v45rj-docker-20.10.12/bin/docker
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ /nix/store/r50lif10qfix73c3rxvjbmsd1x8v45rj-docker-20.10.12/bin/docker build -f Dockerfile --no-cache -t swh/stack .
Sending build context to Docker daemon     99MB
Step 1/13 : FROM python:3.7
 ---> ad37de9b03ef
...
  • Check the scheduler state after docker up:
Jan 24 2022, 11:33 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont changed the status of T3746: staging: Deploy maven indexer/lister/loader, a subtask of T1724: Maven Central repository support, from Open to Work in Progress.
Jan 24 2022, 11:32 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T3746: staging: Deploy maven indexer/lister/loader from Open to Work in Progress.
Jan 24 2022, 11:32 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont triaged T3874: staging: Analyze result of the maven listing and ingestion as Normal priority.
Jan 24 2022, 9:28 AM · Maven loader, Maven lister, Archive coverage
ardumont updated the task description for T3746: staging: Deploy maven indexer/lister/loader.
Jan 24 2022, 9:25 AM · Maven loader, Maven lister, System administration, Archive coverage
ardumont moved T3746: staging: Deploy maven indexer/lister/loader from Backlog to Weekly backlog on the System administration board.
Jan 24 2022, 9:23 AM · Maven loader, Maven lister, System administration, Archive coverage

Jan 21 2022

borisbaldassari updated the task description for T1724: Maven Central repository support.
Jan 21 2022, 4:09 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D7014: Fix comment in doco.yml.
Jan 21 2022, 4:03 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 17 2022

ardumont updated the task description for T1724: Maven Central repository support.
Jan 17 2022, 3:00 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 15 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

@ardumont I've added a nginx container to the main docker-compose file and made it serve one of the example fld files (in the conf/maven-index directory).
The served file can be accessed from the lister container, but for now the task doesn't pick anything -- I don't see it in the lister container logs at all, and (thus) the psql commands returns 0 rows. I'll investigate why (I made it work a month ago, so..), but a quick discussion about the scheduler might help on IRC. I'll be connected on IRC this monday, if we can take a chance to discuss the issue (and check that the compose thing is ok) that would be helpful.

Jan 15 2022, 8:21 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
borisbaldassari added a revision to T1724: Maven Central repository support: D6955: maven: add index nginx host in docker dev setup.
Jan 15 2022, 8:06 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 10 2022

borisbaldassari added a comment to T1724: Maven Central repository support.

Thanks! You did well, I had not been notified about your post and didn't know about it. Sorry for overlooking that. I'll have a look this week.
Happy new year btw, talk to you soon!

Jan 10 2022, 8:39 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Jan 7 2022

ardumont added a comment to T1724: Maven Central repository support.

@borisbaldassari Hello, gentle ping about ^

Jan 7 2022, 4:01 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont updated subscribers of T3835: staging: Ingest sourceforge cvs origins.

@stsp as mentioned in the main task, it'd be neat if you could create yourself an account in sentry [1].
Ping me when it's done so i can invite you in the #swh-team so you can browse the cvs loader
related issues from there.

Jan 7 2022, 3:57 PM · CVS loader, Archive coverage
ardumont closed T3788: staging: Deploy cvs loader v0.1 as Resolved.
Jan 7 2022, 3:50 PM · System administration, CVS loader, Archive coverage
ardumont closed T3788: staging: Deploy cvs loader v0.1, a subtask of T3691: Implement CVS loader, as Resolved.
Jan 7 2022, 3:50 PM · CVS loader, Archive coverage
ardumont added a parent task for T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects: T3835: staging: Ingest sourceforge cvs origins.
Jan 7 2022, 3:40 PM · CVS loader, Archive coverage
ardumont added a subtask for T3835: staging: Ingest sourceforge cvs origins: T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects.
Jan 7 2022, 3:40 PM · CVS loader, Archive coverage
ardumont moved T3788: staging: Deploy cvs loader v0.1 from in-progress to deployed/landed/monitoring on the System administration board.
Jan 7 2022, 3:39 PM · System administration, CVS loader, Archive coverage
ardumont triaged T3835: staging: Ingest sourceforge cvs origins as Normal priority.
Jan 7 2022, 3:38 PM · CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Jan 7 2022, 3:37 PM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Jan 7 2022, 3:29 PM · System administration, CVS loader, Archive coverage

Dec 17 2021

ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 17 2021, 5:06 PM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 17 2021, 4:46 PM · System administration, CVS loader, Archive coverage
ardumont added a revision to T3788: staging: Deploy cvs loader v0.1: D6863: Activate sentry for swh.loader.cvs.
Dec 17 2021, 4:44 PM · System administration, CVS loader, Archive coverage
ardumont updated subscribers of T1724: Maven Central repository support.

On second thoughts: in order to run the docker-dev setup, I also had to run a virtual
machine alongside the swh setup to host the text index file, and make sure the swh vm
could access it. I suppose that any vm/docker/baremetal machine with an apache/nginx
server could do for that, as long as the lister can http-fetch the .fld file.

Dec 17 2021, 3:58 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont changed the status of T3788: staging: Deploy cvs loader v0.1, a subtask of T3691: Implement CVS loader, from Open to Work in Progress.
Dec 17 2021, 3:49 PM · CVS loader, Archive coverage
ardumont changed the status of T3788: staging: Deploy cvs loader v0.1 from Open to Work in Progress.
Dec 17 2021, 3:49 PM · System administration, CVS loader, Archive coverage
ardumont added a project to T3788: staging: Deploy cvs loader v0.1: System administration.
Dec 17 2021, 3:49 PM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 17 2021, 3:49 PM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 17 2021, 3:40 PM · System administration, CVS loader, Archive coverage
ardumont closed T3811: archive.s.o: change Debian tooltip to include derivatives, a subtask of T2400: Ingest current and historical Ubuntu releases, as Resolved.
Dec 17 2021, 2:51 PM · System administration, Debian loader, Package Loader, Archive coverage
vlorentz updated the task description for T1724: Maven Central repository support.
Dec 17 2021, 9:49 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage
vlorentz updated the task description for T1724: Maven Central repository support.
Dec 17 2021, 9:49 AM · Maven loader, Maven lister, GSoC 2019, Archive coverage

Dec 16 2021

ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 16 2021, 5:16 PM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 16 2021, 5:15 PM · System administration, CVS loader, Archive coverage
stsp added a comment to T3692: Stabilize svn loader implementation.

This PDF contains my SVN Loader review report.

Dec 16 2021, 3:58 PM · SVN Loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 16 2021, 11:39 AM · System administration, CVS loader, Archive coverage
ardumont added a revision to T3788: staging: Deploy cvs loader v0.1: D6851: Deploy and activate swh-worker@loader_cvs on staging workers.
Dec 16 2021, 11:30 AM · System administration, CVS loader, Archive coverage
ardumont added a revision to T3788: staging: Deploy cvs loader v0.1: D6849: swh.loader.cvs.tasks: Fix parameter uses to the ones needed.
Dec 16 2021, 10:51 AM · System administration, CVS loader, Archive coverage
zack renamed T2400: Ingest current and historical Ubuntu releases from Ingest current and history Ubuntu releases to Ingest current and historical Ubuntu releases.
Dec 16 2021, 10:36 AM · System administration, Debian loader, Package Loader, Archive coverage
ardumont closed T3798: Debian package for swh.loader.cvs, a subtask of T3788: staging: Deploy cvs loader v0.1, as Resolved.
Dec 16 2021, 9:27 AM · System administration, CVS loader, Archive coverage
ardumont closed T3798: Debian package for swh.loader.cvs as Resolved.
Dec 16 2021, 9:27 AM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T3788: staging: Deploy cvs loader v0.1.
Dec 16 2021, 9:26 AM · System administration, CVS loader, Archive coverage
ardumont moved T3798: Debian package for swh.loader.cvs from in-progress to deployed/landed/monitoring on the System administration board.
Dec 16 2021, 9:25 AM · System administration, CVS loader, Archive coverage
ardumont changed the status of T3798: Debian package for swh.loader.cvs, a subtask of T3788: staging: Deploy cvs loader v0.1, from Open to Work in Progress.
Dec 16 2021, 9:25 AM · System administration, CVS loader, Archive coverage
ardumont changed the status of T3798: Debian package for swh.loader.cvs from Open to Work in Progress.
Dec 16 2021, 9:25 AM · System administration, CVS loader, Archive coverage
ardumont added a project to T3798: Debian package for swh.loader.cvs: System administration.
Dec 16 2021, 9:25 AM · System administration, CVS loader, Archive coverage
ardumont added a comment to T3798: Debian package for swh.loader.cvs.

This has been done though the debian improvment tooling task [1].
Jenkins builds and uploads debian package as usual after a git tag on the repository.

Dec 16 2021, 9:25 AM · System administration, CVS loader, Archive coverage

Dec 13 2021

ardumont closed T2400: Ingest current and historical Ubuntu releases as Resolved.
Dec 13 2021, 11:24 AM · System administration, Debian loader, Package Loader, Archive coverage
ardumont updated the task description for T2400: Ingest current and historical Ubuntu releases.
Dec 13 2021, 11:24 AM · System administration, Debian loader, Package Loader, Archive coverage
ardumont updated the task description for T2400: Ingest current and historical Ubuntu releases.
Dec 13 2021, 11:23 AM · System administration, Debian loader, Package Loader, Archive coverage
ardumont triaged T3798: Debian package for swh.loader.cvs as Normal priority.
Dec 13 2021, 10:30 AM · System administration, CVS loader, Archive coverage
ardumont updated the task description for T2400: Ingest current and historical Ubuntu releases.
Dec 13 2021, 9:43 AM · System administration, Debian loader, Package Loader, Archive coverage

Dec 10 2021

stsp added a comment to T3691: Implement CVS loader.

I found one additional problem. See D6823.

Dec 10 2021, 3:01 PM · CVS loader, Archive coverage
stsp added a revision to T3691: Implement CVS loader: D6823: validate input paths in the CVS loader.
Dec 10 2021, 3:00 PM · CVS loader, Archive coverage
ardumont added a comment to T3691: Implement CVS loader.

Unless I have overlooked something, all currently known issues have now been addressed.

Dec 10 2021, 1:13 PM · CVS loader, Archive coverage
stsp added a comment to T3691: Implement CVS loader.

Unless I have overlooked something, all currently known issues have now been addressed.

Dec 10 2021, 11:40 AM · CVS loader, Archive coverage

Dec 9 2021

ardumont updated the task description for T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects.
Dec 9 2021, 4:11 PM · CVS loader, Archive coverage
ardumont updated the task description for T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects.
Dec 9 2021, 4:11 PM · CVS loader, Archive coverage
ardumont placed T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects up for grabs.
Dec 9 2021, 4:07 PM · CVS loader, Archive coverage
ardumont triaged T3789: Adapt sourceforge lister to list cvs origins according to what the cvs loader expects as Normal priority.
Dec 9 2021, 4:07 PM · CVS loader, Archive coverage
ardumont triaged T3788: staging: Deploy cvs loader v0.1 as Normal priority.
Dec 9 2021, 4:04 PM · System administration, CVS loader, Archive coverage