Sorry for the delayed response.

Sorry for the delayed response. I'm assuming we'd like it better if the lister continued anyway in case of a "fatal" connection error, with maybe some sort of retry?

It finally stopped, albeit poorly, the remote closed the connection.

Still running:

It no longer stops on unexpected 404 ;)

1.3.1 [1] packaged and deployed on staging worker

It broke with the following, sentry should have more detail [1]

May 07 15:57:03 worker0 python3[230930]: [2021-05-07 15:57:03,547: ERROR/ForkPoolWorker-4] Task swh.lister.sourceforge.tasks.FullSourceForgeLister[ec00c8cd-ff5b-47df-adfd-a8c1884b9831] raised unexpected: HTTPError('404 Client Error: Not Found for url: https://sourceforge.net/rest/p/fci-cu-library2/b396')

Deployment in progress.

I'll adapt (but afk).

Check everything is fine (it is):

psql service=admin-staging-swh-scheduler                                                      psql (12.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

Update the scheduler backend with the new task type:

Installing the latest package on a worker:

Note that it helped yet but i reproduced the issue in jenkins *locally*.
Prior to that, other issues with our moving cogs (swh.core, etc...) prevented it
(other unrelated failures arose).

It starts well. The build is stuck [1]

It looks like there are projects outside of the /p/ namespace. Just looking at the very first sitemap, I got an /adobe/ namespace (https://sourceforge.net/rest/adobe/manjobi), which implies that we should also consider namespaces outside of /p/ when listing.

Here's a blueprint for implementing a SourceForge lister, based on an exchange with a SourceForge tech contact:

The scripts and data at https://github.com/chpwssn/sourceforge-items/ look to be exactly what is required with that person (chpwssn) having already identified over 350,000 SVN, Mercurial, and GIT repositories on SourceForge with associated rsync commands for downloading them.

I forgot that diff. This is clearly not the good approach to list sourceforge projects (as a lot of them are missing from the rsync mirrors, which only backup release files hosted on sourceforge), so let's abandon it.

This needs more work as numerous sourceforge projects are missed with the current implementation. I removed that diff from the review queue until I find a better solution.

Below are some intels I managed to gather in order to fulfill that task.