Build is green

Rebase

It's now available on https://annex.softwareheritage.org/public/dataset/license-blobs/2022-04-25/

Experiments in terms of space:

https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4658#note_12824 is the process I've followed for a (lower stakes) removal of buggy raw_extrinsic_metadata messages from swh.journal (in staging).

Build has FAILED

• Use list comprehension for side_effect
• Trigger rebuild

Build has FAILED

feat: Add edition and build time in package metadata

Build is green

• Squash commits
• Improve comments, fix vars, and add docs

Build is green

• feat: Make the lister incremental and use build_time as release date
• tests: Refactor to use fewer lines of code and improve readability

swh.loader.git 2.1.0 has now been deployed on all workers.

neat

I've tested this on all the origins that triggered https://sentry.softwareheritage.org/share/issue/d04e5d7050cb49c6a080b0552fdee5ef/ (with D7838 applied to test on the real storage) and they load fine now.

Build is green

More consistent commit message

Build is green

minor improvments

Build has FAILED

Build has FAILED

Build has FAILED

Looks good to me.

Build is green

clamp before logging

Build is green

Build is green

rebase

fix opam tests

Maybe worth @douardda or @marmoute having a look at this, as I think there were more usecases than just disarchive for this?

Thanks; I have one comment inline that I think should be addressed!

Public review successfully validated, ready for production :)

Build has FAILED

Build has FAILED

In D8812#229076, @anlambert wrote:

Would not it be more generic to validate the listed origin URLs at the scheduler level, more precisely in the record_listed_origins method ?

rebase

validate in the base class

Would not it be more generic to validate the listed origin URLs at the scheduler level, more precisely in the record_listed_origins method ?
This way, all listers will benefit from URLs validation without touching their code.
Proceeding like this, we could even compute the list of rejected URLs in the send_origins method of base lister class (to log a warning for instance).

In D8808#229043, @anlambert wrote:

LGTM, I guess we should deploy this as soon as possible, right ?