Build is green

Update: Prefer to use an extid manifest instead of simply using sha256 checksum as it seems more robust.

migrated to gitlab MR https://gitlab.softwareheritage.org/infra/swh-sysadmin-provisioning/-/merge_requests/88

Build is green

Looks good to me, thanks !

also add rabbitmq marker in pytest.ini

Build is green

Do not json.loads already deserialized json data

migrated to https://gitlab.softwareheritage.org/infra/puppet/puppet-swh-site/-/merge_requests/533

Build is green

Address @vlorentz comments

Thanks!

Build is green

add logging

I just did a proper review now, it still looks good :)

Build is green

Update: Simplify implementation using classes (thanks to @vlorentz)

Now depends on https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4658 instead

In D8798#228725, @vlorentz wrote:

Thanks, I like it.

Before going into the details, what do you think of D8800? it's a refactoring of this diff, using an abstract OriginSaveWebhookReceiver class, and each forge defines a subclass. This replaces calls to origin_save_webhook_receiver with lots of arguments and decorators. I didn't touch the tests at all.

(detaching this task from the Cassandra task tree, because it's getting really big)

(detaching this task from the Cassandra task tree, because it's getting really big)

New objects with duplicated entries go through this method to be fixed: https://docs.softwareheritage.org/devel/apidoc/swh.model.model.html#swh.model.model.Directory.from_possibly_duplicated_entries (currently only used by swh.storage.backfill); but old kafka messages still have duplicate entries, causing the crash above.

Thanks, I like it.

In D8569#228509, @anlambert wrote:

In D8569#228493, @franckbret wrote:

In D8569#228487, @franckbret wrote:

Rubygems: Improve loader to make use of artifacts and rubygems_metadata provided by the lister extra_loader_arguments

Use artifacts and rubygems_metadata to get list of versions, artifacts checksums and extrinsic metadata url
Add an EXTID manifest
Add metadata from extrinsic metadata

@anlambert Please note I used 'rubygems_metadata' instead of 'rubygem_metadata' as in the lister. Maybe I'm wrong but I think the lister should rename to rubygems_metadata?

@franckbret, I did not use plural because we are processing a single gem in the loader (with multiple versions but those are metadata for a single gem).
So I do not think we should modify the lister output.

Build is green

Build is green

Rebase

Update: Address @ardumont and @vlorentz comments

It looks like the changes fail because CI doesn't know to use the model updates in D8759.

one non-blocking suggestion to explicit the compressed nature of the dump inline.

Possibly relevant for the Azure storage: https://learn.microsoft.com/en-us/rest/api/storageservices/find-blobs-by-tags