Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.

Note: when this is (reasonably) done, we should document the addition of SourceForge to the archive coverage page at archive.s.o and also to the archive changelog.

It's deployed and the ingestion is ongoing.
Monitoring of the ingestion will be moved to a dedicated task [1]
Closing this now.

Update for the Maven Indexer prototype: it works! (finally)

Some more information about the maven indexer. Beware people it's a bit dirty, and you're not going to like it infra-wise.

So, to sum up the options we have.. Basically we "just" need all artifacts coordinates. From there for each artifact we can:

Few more cents in the bucket..

• scrapping is explicitly forbidden, see https://repo1.maven.org/terms.html -- however making contact first will help us go through most of the abuse-limiting rules I guess.
• regarding fasten, there are indeed some bits that could be useful. However most of our difficulties are in getting a list of projects, whereas this information is already provided by the user in the case of fasten. So, interesting and useful, but not a game changer regarding the difficult part of our job.

In T3311#64737, @vlorentz wrote:

I think the only issue with (3) is not being retroactive

In T3235#63002, @libEqualizer wrote:

You are likely doing a git pull on a periodic basis. Just add git bug bridge pull [<name>] next to it.

However, this would require considerable work

Hi, thanks for the suggestion.

awesome, thanks @joenio ! you can also drop by our other devel communication channel if you want to discuss this in other ways: https://www.softwareheritage.org/community/developers/

Thanks @zack for the info, I'll start learning the SWH dev stack following the instructions I found in the wiki[1].

Hey, yes, we want to have one, but nobody is working it at the moment, and we rather have someone knowledgeable with that ecosystem to work on it. So, if you're interested, you're more than welcome to help there! (And thank you in advance.)

Hi SWH devs,

After recent exchanges with @hboutemy and Charles Sabourdin, here is a clarification of the scope of this task.
We need a Maven repository lister that addresses the following issues:

Listing deployed in production:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Everything went fine:

worker1.internal.staging.swh.network: Mar 15 10:37:01 worker1 python3[2277003]: [2021-03-15 10:37:01,800: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b]
worker1.internal.staging.swh.network: Mar 15 10:37:06 worker1 python3[2277067]: [2021-03-15 10:37:06,017: INFO/ForkPoolWorker-4] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b] succeeded in 4.2116
25785101205s: {'pages': 5, 'origins': 64}

Checking in staging first, with:

@hboutemy : I wonder if you are aware that we have now in place a grant program that allows to fund development of listers like this one.
All the information is available at https://www.softwareheritage.org/grants and you can mail me for more info if needed.

We would like to see this in prod as soon as reasonably possible.

There is also the VideoLAN Gitlab instance (that will replace the cgit forge) to archive located at https://code.videolan.org/.

new listers

In T376#58337, @ardumont wrote:

Note that does not mean this is or will be ingested anytime soon though.
We are still missing at least the one cog to actually schedule those listed origins.

More details in T2345#58247

Note that does not mean this is or will be ingested anytime soon though.
We are still missing at least the one cog to actually schedule those listed origins.

Here's my understanding of the status of the migration to the next generation scheduler as of today:

Instance cgit scheduled [1]

In T376#57824, @rdicosmo wrote:

Thanks @ardumont , that's great! If you think this does not need any more support on the Eclipse side, may you let them know?

Thanks @ardumont , that's great! If you think this does not need any more support on the Eclipse side, may you let them know?

With the latest improvment, we listed the instance in one request [1]

The 500 seems normal

yes, agreed.

Thanks @ardumont for experimenting with this. The 500 seems normal: we need to tell Eclipse about us first, I'll put you in touch. So maybe it's still a no-brainer, and we just need to document the "contant the owner to get whitelisted" human step :-)

In the context of deploying the next gen lister in staging (T2998), i also tried the eclipse cgit instance

Now that we have a cgit lister, this should be a no brainer.
If that's the case, we need it up and running quickly.

• "'outdatedest' origin": excluding disabled origins and origins visited after their last_activity (if any), the min(current_time - last_visit) (lower is better)

thanks, looks a good starting point.

• "origins with pending changes": Number of origins where last_visit < last_activity (lower is better)

Some potentially interesting and "easy" metrics:

great, thanks! :)

It was (and thanks for the ping).

was this completed? i suspect this issue can be closed if the gitweb is being regularly crawled.

All instances listed in this task have been added, so I'm closing this. Other instances can be added in the future, submitting matching task for tracking reasons as needed.

I've completed a bunch of these in 86f8b213e23970feb9f9bda8ab87fc7d6851abf0