Rebase

Add explicit test for default behavior

In D8914#231729, @anlambert wrote:

Thanks, I will try to update the swh/lister/rubygems/data/small_rubygems_dump.sh script to generate a postgres dump that can be loaded when ERROR_ON_STOP is set.

...

Thanks!

(I've at least pushed the main branch and relevant tags)

As I've suspected, this is due to the pristine-tar branch having objects that exceed 100MB:

As mentioned by @ardumont, please bump the schema version and add a migration script.

To force kafka compaction to run I've done the following:

LGTM too, thanks!

Thank you!

In D8386#229890, @anlambert wrote:

In D8386#229882, @olasd wrote:

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

This seems like a misfeature in the webapp:

https://archive.softwareheritage.org/api/1/snapshot/158a3f36b0bd3da461fb7458de44cfa2c94e4270/

The snapshot has multiple branches, with the same version suffix, pointing at the same objects (because the exact same version of the package is present in multiple Ubuntu suites).

I'm not 100% sure how we should be fixing that, but that bug shouldn't prevent you from giving the fedora snapshots the "semantically correct" structure.

I also noticed that yesterday evening and I was also wondering what is the best way to fix that. I see two possible options:

1. We change the names of the keys in snapshot branches dictionary in order to use the intrinsic version of a debian package instead of its extrinsic one (meaning releases/bionic-security/main/1.14.0-0ubuntu1.10 should rather be releases/1.14.0-0ubuntu1.10)
2. We update the webapp to filter duplicated releases before display as the release name is used instead of the snapshot branches key associated to the release

I would rather go for the second one as keeping the debian/ubuntu suites and components in keys of snapshot branches dictionary seems of interest.
We could do the same for the fedora case as based on my tests it is quite common that extrinsic versions in the form [0-9].[0-9].[0-9]-[0-9].fc[0-9]+
target the same intrinsic version [0-9].[0-9].[0-9]-[0-9].

In D8386#229677, @KShivendu wrote:

@anlambert

I noticed that https://archive.softwareheritage.org/browse/origin/directory/?origin_url=deb://Ubuntu/packages/nginx has duplicate branch names, which is very confusing. In fact, even the default branch is repeated twice and I see two check marks. If we use branch names like 0.3.9-15.fc26, won't the same happen with Fedora listers? It doesn't seem to differentiate between the editions. (or does it?)

Thanks!

I'm not too sure about having a single "big" object as json output, as dumping that full json structure will only work out if the process didn't crash in the middle (and when that does, you won't really know what had happened until then). Maybe output one json object per project mutated instead?

Looks great, thanks.

Looks nice!

Experiments in terms of space:

https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4658#note_12824 is the process I've followed for a (lower stakes) removal of buggy raw_extrinsic_metadata messages from swh.journal (in staging).

swh.loader.git 2.1.0 has now been deployed on all workers.

I've tested this on all the origins that triggered https://sentry.softwareheritage.org/share/issue/d04e5d7050cb49c6a080b0552fdee5ef/ (with D7838 applied to test on the real storage) and they load fine now.

More consistent commit message

Maybe worth @douardda or @marmoute having a look at this, as I think there were more usecases than just disarchive for this?

Thanks; I have one comment inline that I think should be addressed!

In D8808#229043, @anlambert wrote:

LGTM, I guess we should deploy this as soon as possible, right ?

LOL, thanks Jenkins

Apply @anlambert's suggested changes

Thanks!