Rebase

If we want to remove perms normalization from the tarball loader, then we need to discuss that in a separate task. This fixes and introduces tests for the behavior that was intended by the original code, which was buggy.

D5652 adds tests for the manifest format, so I don't think you need to do that here.

Don't forget to drop the notice calls ;)

Please mount the shared cache directory in the image to avoid re-downloading all of PyPI from the internet on every build. I'd suggest duplicating the existing includes/agent-docker.groovy.j2 to a new agent-docker-sphinx.groovy.j2 file, to add the options relevant for this build (for instance, I don't know how useful mounting a tmpdir on /tmp would be).

Now deployed in prod:

New swh.loader.core deployed in staging.

In D5585#141819, @ardumont wrote:

I'd probably keep the new package repo instructions in here as they have no counterpart in the developer docs; Move that to the bottom, maybe?

I think this draft is about complement this ^ and needs to be reworked and merged into an actual doc [1]

And there is a dedicated task for it btw [2]

[1] https://hedgedoc.softwareheritage.org/CaqENEY3RJi19idS8MggWw

[2] T3179

I'd probably keep the new package repo instructions in here as they have no counterpart in the developer docs; Move that to the bottom, maybe?

In T2602#63432, @vlorentz wrote:

For the harder cases, that involve changes to the PK, we could do something like this:

• create a new table with a new name (eg. revision_v[n+1]; like we do in swh-search except Cassandra does not support aliases)
• start an extra storage backend, that reads from that table instead of the old one (eg. revision_v[n]), and also reads from all the other tables as usual
• have a multiplexing storage proxy (like we have for the objstorage), that queries this new backend (which reads from v[n+1]), and falls back to the old backend (which reads from v[n])

Some partitions have reached the tail of the journal and everything is still running smoothly, yay.

So D5246 has landed a while ago. The s3 object copy process has now caught up on some partitions and I can confirm that the copy of the latest added objects happens without any race condition.

Sure, you can keep this implementation, that's why the diff was accepted in the first place.

Could you deduplicate swh_scheduler_peek_any_ready_priority_tasks and swh_scheduler_peek_tasks_with_priority?

Very, very nice! Thanks.

Please fix the coding-guidelines ref as well.

I don't understand why this is needed. Aren't we be able to explicitly send instances of the existing swh.loader.git.tasks.UpdateGitRepository task to a separate queue, and have a celery process consume the "regular" tasks from that queue directly?

The process has been restarted and is well ongoing (we have 800 million objects left to copy, at around 500 ops, so the ETA until reaching the tail of the log is around 3 weeks now).

Knobs to adjust the visibility of origins in the archive and in the web API

In T3087#63054, @vsellier wrote:

Are we planning to add a way to notify the mirrors of the takedown notices ?

I'm sure some of the assumptions behind this script are going to break, eventually. But for now it's better than nothing!

I'd rephrase the suggestions to "Please install 'swh.model[cli]' for full functionality.", rather than try to guess what the user did. They likely got the swh-identify script from installing swh.model as a dependency of something else!

Jenkins runs the py3 tox environment explicitly, so this won't get run by CI. The same issue exists with the previously added "identify" environment.

And this is now available in production.

This has now been deployed and tested in staging with a canary origin (github.com/olasd/Pythagore). Time to deploy in production.

Operationally, there's two axes we can play with:

(I've also noticed dry_run was = True, so I fixed that as well :P)

Tail of log:

The script is now running on getty.

Add comment for the sha512 field

The migration script has now run to completion (took around a week).

I've deployed the extid schema changes on all storages, and I've started the migration script on getty.

+1 from me, this guide is great.

Not quite sure about the split between BasePackageInfo / BaseManifestPackageInfo (and I really don't like the new name).

The following objects remain:

Shouldn't the extid() methods all return a tuple (extid_type, extid_value) rather than a plain bytes value? I can imagine a point where, for the same loader, we might want to change the extid_type, and the current implementation wouldn't be able to distinguish them.

After a lot of back and forth, and the release of swh.model v2.3.0 and swh.storage v0.26.0, this is now all done and deployed in staging and production.

After the release of swh.model v2, this is now done.

The missing topic (raw_extrinsic_metadata) has been handled as part of the migration in T3019. Closing.