Here is an overview of the fields (+ internal version name + branch name) used by each package loader, after D6616:

The main loader is now deployed.
Another task got opened to keep the references on the possible improvments [1]

FTR, olasd, douardda and I discussed an inconsistency in keys used in kafka, and decided to use hashes for all origin/visits/visit statuses; and doing the same for ext metadata in both kafka and the DB solves the issue about defining unicity.

@rdicosmo a full example of what?

The suggestion was to have extrinsic metadata on directories that come from a deposit of a bundle (e.g. .tar.gz or .zip file coming from HAL), instead of on a synthetic revision as is currently the case, so they can be accessed knowing the hash of the directory (which is an intrinsic id).

Alternatively, we could keep writing the metadata on revision/releases, and use the provenance service (when it's ready) to find them from a directory SWHID. What do you think?

I'm currently using the following regex to filter the exposed urls .tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$ but I'm pretty sure it could be improved.

If that ever happens again (I don't expect it but I have been wrong in the
past), we will have a logged exception with some context.

Build is green

Build is green

rework commit message again (last time!)

• Adapt according to review (log.exception, prefer unexpected to divergent)
• Rework commit message to align with the diff

D2949 updated accordingly

Build is green

Drop unnecessary log instructions

Build is green

• Fix existing source.json which should not have been modified in the first place
• Add test scenario around the modification the diff introduces

Build has FAILED

Rebase on latest master

It's not invalid in the end, we hit it in production now.

and done:

May 25 17:17:10 worker01 python3[603377]: [2020-05-25 17:17:10,297: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[5be5167d-3783-4fc9-b18e-116424949d0a] succeeded in 3153.633277214016s: {'status': 'eventful', 'snapshot_id': '76f38fb09efe9f10305c50f5cf083b008217256f'}

Even simpler than D2949, filtering out the culprit branch "evaluation" (targetting nixpkgs) and we are back on track.

Replaced by D3178

Build has FAILED

• nixguix: Add a test around the wrong metadata revision structure detection

Build is green

Rework according to review
(still working on a test)

Working on adding one test which pass in there (load from archive loader, then load from nixguix whose listing list the same artifact...)

I actually don't hit the issue in a test...

Well, i did through patching...

Build is green

Rework commit message

Build is green

• Rebase on latest master
• when a missing key is triggered, continue checking for other revisions of the snapshot

Working on adding one test which pass in there (load from archive loader, then load from nixguix whose listing list the same artifact...)

Working on adding one test which pass in there (load from archive loader, then load from nixguix whose listing list the same artifact...)

It's not invalid in the end, we hit it in production now.