In D8539#222941, @olasd wrote:

In D8539#222800, @ardumont wrote:

Fine, i've one comment i'd like others to have a look at though [1] regarding where
that new discovery (interface) code should go. It feels currently a bit off to me that this code
is in loader-core. Loaders are not the sole archive consumers (scanner, webapp, cli, indexer, cooker, ...).

[1] https://forge.softwareheritage.org/D8539?id=30916#inline-60774

@vlorentz @douardda @olasd @anlambert ^

If it's to be used generically, this discovery code should pretty definitely not be in swh.loader.core.

• The generic discovery algorithm, and base abstract classes/protocols, should probably be in swh.model, as they're tied to that structure;
• The swh.storage-based discovery mechanism could live in swh.storage.algorithms, and be used by swh.loader.core;
• The REST API-based discovery mechanism could live in swh.web.client, or stay in swh.scanner.

In D8909#231631, @vlorentz wrote:

@anlambert Shouldn't this be replaced by swh auth generate-token?

Some screen capture of the differents scenario when login:

Simplify some code and rebase

@anlambert hi, can we merge this one?

Rebase

Check the number of http requests done in incremental tests

@KShivendu Hi, we had a similar situation with rubygems loader, see its _load_directory method

Improve test for incremental listing, ensure the http searchQuery/lastUpload value is a is a date

In D8663#229574, @vlorentz wrote:

buuuut you are using a strict inequality, so you need to subtract one day, in order not to miss uploads submitted after the previous run of the lister but on the same day.

Also, you should apply .astimezone(tz=timezone.utc) before converting to date, because the database is not guaranteed to return timestamps in UTC even when they were written in UTC.

(Sorry for the back-and-forth; hopefully I'm done now.)

Use greater than or equal instead of strict comparison when building http api query params for incremental listing

Abandon revision because in this case we can not really get advantages of an incremental mode

In D8824#229544, @anlambert wrote:

@franckbret, as explained in my inline comment we cannot use the date filtering on the release index of CPAN elasticsearch.

The only incremental mode we can implement here is to filter the ListedOrigininstances sent to the scheduler according to the
last_updatevalue, if it is greater than the date from the lister state, we can yield it.

Nevertheless, I am not sure if it is worth it as a full listing takes around 10 minutes, which is pretty fast.

Rebase

Rebase

minor improvments

Use an offset of -15h when querying the api which is the lower timezone recorded in the tzdb

Ensure we query the api with the same timezone Us/Pacific date as the http api use for querying and expressing results

Do not json.loads already deserialized json data

In D8569#228509, @anlambert wrote:

In D8569#228493, @franckbret wrote:

In D8569#228487, @franckbret wrote:

Rubygems: Improve loader to make use of artifacts and rubygems_metadata provided by the lister extra_loader_arguments

Use artifacts and rubygems_metadata to get list of versions, artifacts checksums and extrinsic metadata url
Add an EXTID manifest
Add metadata from extrinsic metadata

@anlambert Please note I used 'rubygems_metadata' instead of 'rubygem_metadata' as in the lister. Maybe I'm wrong but I think the lister should rename to rubygems_metadata?

@franckbret, I did not use plural because we are processing a single gem in the loader (with multiple versions but those are metadata for a single gem).
So I do not think we should modify the lister output.

In D8569#228487, @franckbret wrote:

Rubygems: Improve loader to make use of artifacts and rubygems_metadata provided by the lister extra_loader_arguments

Use artifacts and rubygems_metadata to get list of versions, artifacts checksums and extrinsic metadata url
Add an EXTID manifest
Add metadata from extrinsic metadata

Rubygems: Improve loader to make use of artifacts and rubygems_metadata provided by the lister extra_loader_arguments

Ok, thanks for the feedback, will have a look on this next week, I will be out of office until wednesday.

Rebase

In D8762#227816, @anlambert wrote:

Looks good to me, I guess you need to update the loader too.

Improve documentation section related to incremental listing

fix issue with hexadecimal representation of the checksum md5 hash

Manage case where author or last_update is empty

Remove description from message

In D8566#227169, @franckbret wrote:

@anlambert Can we merge this one, do you have more feedback?

@vlorentz Can we merge this one?

@anlambert Can we merge this one, do you have more feedback?

Get md5 checksums from HEAD request on archive url

Remove original-artifacts-json from raw extrinsic metadata, it should already be created by the base package loader

Pub.dev have no other metadata we do not already parse.

@vlorentz @anlambert We can get archive checksums by making an HEAD call to archive download url :

loader specification documentation, update Hackage section

Remove 'pubdev-pubspec-json' format from raw_extrinsic_metadata

Incremental operations are now related to the last_listing_date

In D8663#225586, @vlorentz wrote:

for entry in page:
    last_update = iso8601.parse_date(entry["lastUpload"])
    if not self.earliest_update or last_update > self.earliest_update:
        self.earliest_update = last_update

This makes self.earliest_update the latest lastUpload, not the earliest.

rebase

rebase

rebase

In D8640#225229, @vlorentz wrote:

In D8640#225200, @franckbret wrote:

Remove useless check condition when getttinh 'atuhor' data

You checked there is no empty string in the input data, right?

Remove useless check condition when getttinh 'atuhor' data

Manage checksums

Remove files from commit

CI rebuild

Adapt raw extrinsic metadata tests for CI

Add pytest verbose option to tox (previous failed)

Temporaly add verbose options to pytest to understand why CI fail

Ensure last_update is utc

Add "crates-package-json" raw extrinsic metadata

@vlorentz @anlambert Last commit introduce raw extrinsic metatata with format="original-artifacts-json". It is populated with data from extra loader arguments "artifacts".