This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

Triggered back a run and it seemed to have run well:

Mar 11 12:23:15 worker14 python3[2494505]: [2022-03-11 12:23:15,252: INFO/MainProcess] Received task: swh.lister.cgit.tasks.CGitListerTask[e7674643-c7ca-4ae1-910e-9ea24e8c2335]
Mar 11 12:23:17 worker14 python3[2961235]: [2022-03-11 12:23:17,950: INFO/ForkPoolWorker-7] Task swh.lister.cgit.tasks.CGitListerTask[e7674643-c7ca-4ae1-910e-9ea24e8c2335] succeeded in 2.3799873050302267s: {'pages': 1, 'origins': 177}

Sentry issue: SWH-LISTER-4V

Should be a matter of updating such task with:

Fixed in swh-lister v2.2.0 and deployed to production, closing this.

Build is green

Rebase

Looks good to me !

Build is green

Adapt according to review:

• Add more cgit instance samples
• Rework docstring
• Rework url computation a bit

Build is green

Adapt according to review

The current implementation will not work for most cases where the base cgit URL and the base clone URL differs (see inline comment).

I guess you mean that some cgit origin URLs previously loaded into the archive will not be the same if
the approach in that task is implemented ?

Analyzing further the suggestions using the deprecated swh-lister cache db table as data
point (production data) [1], 3 instances so far will generate sometimes wrong origin
urls with the suggested approach.

Analyzing further the suggestions using the deprecated swh-lister cache db table as data
point (production data) [1], 3 instances so far will generate sometimes wrong origin
urls with the suggested approach.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/400/ for more details.

• Introduce swh-listers fixture in that diff
• Use swh.core.pytest_plugins fixture (requires swh.core >= 0.0.73)

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/387/ for more details.

D1929 took care of it ;)

I'd be inclined to use a composition of solution:

• having the listing policy determined at cgit instance lister initialization (3.)
• Since 1. has already been done in the past, use that instead as a fallback (eclipse and freedesktop might be popular enough to sustain the load for a tad more requests than the current bare listing we do).

http://git.upsilon.cc/ (@zack's git repositories) might be relevant.

Thanks for your interest in working on this @nahimilega , it would be very useful to move forward on a bunch of pending ingestions, including Tor !

In T1659#33593, @anarcat wrote:

but i'd say convert to python. depending on xmllint is very brittle... i already had to tweak the thing once to make it work at all, and the pipeline is kind of nasty. i think you will have to import some HTML parser at some point anyways, so you might as well bite that bullet now.

not that I get a vote in this, but i'd say convert to python. depending on xmllint is very brittle... i already had to tweak the thing once to make it work at all, and the pipeline is kind of nasty. i think you will have to import some HTML parser at some point anyways, so you might as well bite that bullet now.

I would be pretty interested in integrating it with other listers and moving it to the common repo. I guess we can proceed in two ways.

1. Use the script that we already have. Just run this script via python to get the list of repos.

i couldn't find the time to work through the developer setup and the lister tutorial, so I used the shell script to generate a list of projects for tor gitweb.