Use the built_at date instead of publication date

Extract swhid computation

This is the initial plan that I was going for with regards to what to load inside the .gem.

Remove ugly hack

In D8538#222541, @vlorentz wrote:

I still don't understand why you need the _all_contents hack instead of doing this:

async def skipped_content_missing(
    self, skipped_contents: List[Sha1Git]
) -> Iterable[Sha1Git]:
    """List skipped content missing from the archive by sha1"""
    contents = [{"sha1_git": s} s in skipped_contents]
    return (d["sha1_git"] for d in self.storage.skipped_content_missing(contents))

Add docstring, abstract method decorators, explain hack

Use ABC

One question for reviewers: I've tried using Protocol for ArchiveDiscoveryInterface (should I?), but I'm not sure if it's possible to define a concrete method within the protocol, namely if __init__ could be implemented by default in ArchiveDiscoveryInterface.

Rebase

Add assertion for data consistency in tests

Add coverage of random sampling and skipped content

(maybe) fix CI?

Address comments

In D8521#221365, @vlorentz wrote:

There seems to be two chunks in get_sample not covered by tests. Could you add some coverage for them?

Address nits

Fix import

Optimize loops, add types, improve docs and fix a bug

Rebase

Rebase

Rebase

Remove state save when full listing

Add incremental celery task and its test

In D8298#216376, @anlambert wrote:

You also need to add the celery task for the incremental lister in tasks.py

Incremental mode, better testing

Fix infinite loop

Fix loader task

Adapt from lister changes

Rebase

Use pkg.go.dev instead of the proxy

Reflect the changes from the lister

List per package, not per version, prefix with proxy URL

List per package, not per version, prefix with proxy URL

In D8296#216033, @vlorentz wrote:

Hmm... okay then. I guess for reproducibility it's more useful to get the same thing go get than the backend. What do origin URLs look like, though?

Fix release names and update description

Remove useless cached_property and document releases

In D8296#215935, @vlorentz wrote:

Do we really need to use the Go proxy? Can't/Shouldn't we access the real origins directly?

Move to the end and add a note about it being optional

Rebase

Add docstring

Update README, sort setup.py and use f-strings

Looks good to me now. There still remains the question of whether we want to store the yanked info for each release somewhere. Off the top of my head it would make the (upcoming) incremental listing/loading less efficient. Maybe that's out of scope for now. :)

Oops, it seems like this got out of draft too fast. arc says --update and --draft are mutually exclusive for some reason so it's not obvious to me how to tell Phabricator that we still want a diff to stay draft even if the CI passes.

Add test_tasks.py

In D7329#190995, @vlorentz wrote:

LGTM

Remove use of dateutil and add small fixes

Incorporate agreed upon changes

Thanks, how about those?

Add typing and change commit description

Oh, I didn't realize that my approval was needed.

Should this be folded with the patch in T3963#79379?

This looks good, considering the... interesting launchpad API.

D7130 has been committed, so a new version of the loader for faster ingestion will need to be released.