Rebase

Add a fake_pubdev.sh script that generate fake http api response and fake package archives for test purpose

In D8305#216381, @vlorentz wrote:

Thanks, looks good.

One last thing: could you find smaller alternative for the test files? These are quite big, especially packages_pdf_versions_3.8.2.tar.gz

Remove filename from release name

Add documentation

Add pubdev entry to package loader specifications documentation

Testing authors and empty author

Some improvments after review

Some improvments after review

Remove package versions data from lister.

In D8287#215639, @vlorentz wrote:

Hey,

If I understand this correctly:

1. get_pages() first gets a list of packages, then
2. get_pages() fetches all these pages, then
3. get_origins_from_page() creates an origin for each page with the list of versions, with a list of version of that package

Given that the lister's job is only to discover a list of origins, I think the lister should only do step 1 and create a ListedOrigin.
The loader should then fetch the list of versions of each package, which is part of its job.

The only reason some listers fetch the list of versions is that it would not be otherwise available to loaders, but that is not true here, so we should avoid it.

This keeps the architecture closer to what we want to do, and avoids bloating the scheduler database with (potentially long) lists of releases in each row.

Rebase

Rebase

ask @vlorentz for review

Rebase

Rebase on master

Rebase on master

Adapt package loader specifications

In D8259#214944, @franckbret wrote:

Add documentation

Add documentation

Can we merge this one?

Add a test that check various packages names and versions with different syntax

Add aur entry to package loader specifications documentation

Split artifacts to artifacts and aur_metadata.

Rephrase the documentation part about regular and split packages

Fix docstrings errors

Fix documentation build

Document the module

Add a directory cleanup test

Add a finalize method that remove temporary directory once the lister complete

Split artifacts data to artifacts and aur_metadata

Some improvments after code review

Fix typos in docstrings

Change EXTID_TYPE name and MANIFEST_FORMAT

Add 'yanked' information to PackageInfo and adapt tests

Remove http api fixtures from tests data
Adapt fake_crates.sh script accordingly

In D8171#213648, @franckbret wrote:

Switch to manifest for EXTID generation
Remove useless http api call to get the release date

Switch to manifest for EXTID generation
Remove useless http api call to get the release date

In D8171#212616, @ardumont wrote:

Hello, was there a discussion leading to this implementation? If so, it'd be nice to mention it in the description (please).

Nope

Abandonning this one as the same content has been moved to crates lister module docstring ( see D8206 )

Module docstrings need to be the first statement of the file

Fix hyperlink, only the reference to named anchor is needed

Try again to fix the internal hyperlink to a section, this time replace \# with \:

Try to fix internal ref link

Some more typos, replace a link with an internal ref

Rebase

Fix some typos
Rename 'metadata' dict to 'crates_metadata'

@vlorentz @ardumont Here is a proposal to make the crates loader incremental.

Move 'yanked' information from 'artifacts' to 'metadata''
Add some more repository related debug logging

In D8101#211919, @vlorentz wrote:

Don't forget these:

In D8101#211059, @vlorentz wrote:

• Where is last_update used? I don't see it in the artifacts list in the tests
• Finally, could you reword your commit message to be in the imperative mood and mention it is the crates lister? eg. crates: Implement incremental listing

(and this diff's title too, to be consistent with the commit title)

In D8101#211914, @franckbret wrote:

Add 'yanked' to artifacts data

'yanked' is a flag used to determine if a package version has been unreleased

That's not the right place; this format is meant to describe tarballs: https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#original-artifacts-json
but the "yanked" status comes from higher-level objects.

@vlorentz @ardumont Are you ok with this one, can I push?