In T4232#84826, @vlorentz wrote:

note that swh-web currently does it purely on the client side, using the GitHub API. It may be good to move it server-side, though (avoids leaking users' IP addresses to GitHub, and also fixes some failure conditions mentioned in T4055)

note that swh-web currently does it purely on the client side, using the GitHub API. It may be good to move it server-side, though (avoids leaking users' IP addresses to GitHub, and also fixes some failure conditions mentioned in T4055)

@Alphare lgtm

Build is green

Add test_tasks.py

LGTM

Thanks. Should I add a tests_tasks.py? I'm not 100% sure of their usefulness but they're in all other packages.

In D7329#190995, @vlorentz wrote:

LGTM

LGTM

Build is green

Remove use of dateutil and add small fixes

Looks good, thanks.

It's steadily progressing so closing now.

staging lister happily lists new bzr and cvs origins now:

Clean up:

This task has been complete since a while now, closing it.

After some fighting to untangle the mess we had in the scheduling dbs:

• wrong task type used
• wrong data format in old entries

What a mess! The existing data both in staging and production are not in the expected
shape for the loader. Hence the issue of failing the load [1]

In T3032#58243, @anlambert wrote:

Two possible solutions here:

• use stretch/updates, buster/updates, ... as suite names
• add specific URL template processing in debian lister implementation

I would go for the second one.

It's been deployed for a while.

I've duplicated the credentials for the relevant forges, and updated the following instance names:

Still ongoing.

17:31:19 softwareheritage-scheduler@belvedere:5432=> select now(), visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where l.name='gitlab' and l.instance_name='gitlab' group by visit_type order by count(
*) desc;
+-------------------------------+------------+---------+
|              now              | visit_type |  count  |
+-------------------------------+------------+---------+
| 2021-07-29 15:46:55.979469+00 | git        | 1747880 |
+-------------------------------+------------+---------+
(1 row)

It's still ongoing:

Listing ongoing and number of origins increasing regularly. From 200k yesterday [1]
(prior to the deployment of the lister) to ~800k [2] today. Progress.

Deployed v1.5.0 and run triggered and ongoing:

Closing this as requested.

Great, Thanks a lot.

For the record, my lister is still running, 1320500 gitlab.com origins listed so far.

As usual, very nice analysis \o/

It seems the remaining lister instances to process are the phabricator ones that also need credentials.
This is what we currently have in the listers table in scheduler database.

Thanks @ardumont for the heads-up.
As a side note I still have on my to-do to add authentication and svn support to the tuleap lister (but that will happen *after* the maven lister work).

I've updated the listers with no credentials:

Deployed and running so closing.