minor caseness issue

Almost there !
(and thanks a lot for your persistence on this one)

LGTM.

Thanks a lot for this recap Morane !

There are two separate use cases here, so I'll comment on them separately:

Thanks for your interest in working on this @nahimilega , it would be very useful to move forward on a bunch of pending ingestions, including Tor !

resolved (by T691)

In T691#33551, @olasd wrote:

After processing the logs of the backfilling process to make sure to redo all the ranges that were interrupted in various database migrations, I'm now confident that this task is complete: we have a full mirror of all contents on Azure, which is kept up to date by the main archive storage backend writing synchronously to it.

Getting rid of ReCaptcha for save code now LGTM too.
I just wasn't sure that rate limit applies to Web UI submissions (e.g., will API requests come from our own IP? and if so, is that whitelisted?); I'm assuming that is what @anlambert plans to check.

In T1789#33370, @sandipbhuyan wrote:

Can we have the feature which will return the content of File Type, Language Type, and License not its URL

Hi @sandipbhuyan , I had in fact already created a task for this, it's: T1789

btw, the list is ~400 repos for now

@anarcat please hold off from using save code now for now. As we're planning to have a proper cgit lister, we can just add your instance to your rotation once that's done (unless this is super urgent, that is). That will have the additional advantage that we will automatically notice when new repos show up.

It's not really related, because gitweb and cgit are two different things.

Thanks @olasd, @ardumont, and @anlambert for this, it's a great plan and I like it a lot !

Thanks @vlorentz for this first draft. In spite of all the comments above, I think it's a very good start.

The most recent update of the state of this task has shown a regression in the journal test coverage, which, per se, is not a big deal (just a few points). But it does raise the question of how, once we have attained whatever "minimum" coverage we are OK with, we monitor overtime that there is no regression. For instance, I think that code reviews should show to the reviewers how the submitted diff affects code coverage. Ideally, reviewers should be able to so if it has a net positive or negative effect on coverage, and take that into account in their review decisions. (Which is not to say we should never accept diffs that decrease code coverage—there might be reasons to do so. But it is a data point that would be useful for reviewers to see.)

Do we really want to do this?

Just a couple of comments:

• the current proposal is ori instead of org as 3-letter stem
• your use cases are all valid, but would equally work with a full URL and with a hashed URL

In T1731#32712, @vlorentz wrote:

Okay then. I'll work on updating the identifier specification.

So, again, what are the remaining issues that inhibits you to just go ahead and use URI hashes as Cassandra origin IDs?

Those I listed above, which were more "philosophical" than technical. I started implementing it last Monday anyway, and it looks good.

I don't like the idea of this lister.

In T1731#32684, @vlorentz wrote:

In this case, we'll also need to have an identifier for URL + type, if they want to cite/link to the non-default one.
We could use the "contextual information" mechanism, eg. swh:1:ori:SHA1;type=git

In T1731#31900, @vlorentz wrote:

This sounds like a good idea.
But it has some weird implications on components that use the concept of "origin head" (web UI and metadata indexers); because they'll use radically different content depending on which loader visited last.
But having two VCSs at the same URL is weird in itself, so 🤷

[ moving here my feedback from D1516 ]

In D1516#34010, @olasd wrote:

I'm not convinced this is such a good idea; this machine is way more than a "db replica" server (it only has one replica, most its databases are actually primary) and I don't think DNS provides the appropriate granularity level to record this information.

I think the only thing missing here is adding the NPM logo to the archive coverage page.

I think this is now done, right @anlambert ?

only 3% to go in -lister and -core \o/

these catch-all meta-tasks that will grow forever are not terribly useful, the individual tasks + their tasks should be enough

closing this catch-all meta task, the individual sub tasks are clear enough

looks like this is done now, as you're deep in the implementation already! closing

We've discussed this back then, and decided in the end to leave it at the Django layer. Closing.

has this been completed since?

can we haz this, please? :)

snapshot count is now there, closing

is this still the case?

this hasn't happened for a long while

Just checking in on this, are we are discussing moving DBs around. Do we still don't have a backup for the indexer DB?
If so, priority of this one should probably be raised.

This is done, I've forked off the part about consistently documenting configuration options to T1758.

swh-storage-testdata is gone, closing

we have had this for a while now

@douardda: can I punt this to you to either further investigate or just close as Invalid? 3 years later it might no longer be relevant…

closing, we do have an SVN loader now: it has still some issues, but the bulk of the job is done

how many are left? can we close this as well as T419 now that the PyPI listers/loaders have been in production for a while?

@anlambert what's the status of ingesting very large SVN repos, now that we have put the loader in production?