Deployed in production as well and triggered a run:

It's not deployed but actually shipped in the lister v1.4 release onward.

It's not deployed but actually shipped in the lister v1.4 release onward.

From a quick test, it looks like the "Project and release activity details" feed can go back multiple years without any issue, allowing us to backfill the data for all known origins, before adding the incremental behavior to the lister.

Deployed in staging and triggered a run:

This has all landed during the early 2021 sprint.

This has all been landed during the sprint at the beginning of 2021.

Hey @hm, I don't see any typo in https://forge.softwareheritage.org/source/swh-lister/browse/master/docs/run_a_new_lister.rst$50-51. If you do, please make a revision to fix the same.

This got landed through another diff [1] already.

I trust that you made the changes I suggested

I trust that you made the changes I suggested

Build is green

roh, i had forgotten to commit the main subject commit line, fixed!

Build is green

Adapt according to reviews

Let's not make this great piece of doc wait any longer.

lgtm, thanks. a bunch of typo-fixing comments below

Build is green

Small docstring fix

Tagged and deployed v0.9.1

In T3032#58273, @anlambert wrote:

Will you attend to it or should i (i can't tell from your suggestion :)?

I handle it, diff incoming.

Will you attend to it or should i (i can't tell from your suggestion :)?

Two possible solutions here:

• use stretch/updates, buster/updates, ... as suite names
• add specific URL template processing in debian lister implementation

In T3032#58239, @ardumont wrote:

For the debian lister, after updating its task input, i got [1], but i don't really what to do with it yet:

[1] https://sentry.softwareheritage.org/share/issue/7e22ced788c7443093144d69f5707b5e/

The current lister implementations work for the main debian distribution but cannot work for the "debian-security" instance currently.

From the current lister implementation:

base_url = urljoin(self.url, f"dists/{suite}/{component}/source/Sources")

we'd need, for that specific instance:

base_url = urljoin(self.url, f"dists/{suite}/updates/{component}/source/Sources")

see: http://deb.debian.org/debian-security/dists/buster/

• Scheduler cgit lister for instance eclipse

I took the opportunity to unstuck yoctoproject which was in 03/2020 as the next-run-scheduled listing.
It ran with the new lister version and now urls are referenced in the scheduler table listed_origins.

• migration script ran
• restart swh-worker@lister
• disable down cgit instance (kde, happyassassin)

And, first, for cgit listers, prepare a migration script for the existing scheduled
cgit task instances to add the base_git_url to the arguments/kwargs dict [1]

swh.lister v0.8.0 packaged with the latest packagist lister port.

With D4987 on the verge of being packaged, readapted the current listed_origins (staging) with:

swh-scheduler=> begin;
BEGIN
swh-scheduler=> update listed_origins set url=regexp_replace(url, '/$', '') where lister_id='1af42c1b-69b0-41ef-ad98-371843de406e' ;
UPDATE 1340
swh-scheduler=> select * from listed_origins where lister_id='1af42c1b-69b0-41ef-ad98-371843de406e' ;
swh-scheduler=> commit;
COMMIT

(so they match)

Note that with the new cgit implementations, this adds a trailing / at the end of the origin urls.
It's still resolvable at least with our tryouts on either the cgit instances git.kernel.org and git.eclipse.org.

Run ok:

swhworker@worker0:~$ SWH_CONFIG_FILENAME=lister.yml swh lister run --lister cgit url=https://git.eclipse.org/c/ instance=eclipse base_git_url=https://git.eclipse.org/r/

Schedule gnu on staging:

swhworker@worker0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-gnu-full
Created 1 tasks

Build is green

Rebase

Looks good to me !

Build is green