@ardumont I squashed commits and do some cleaning and remove the WIP status.

• package.loader: add origin argument
• Add the functional loader
• cli: add the functional loader in the cli tests

• cli: add the functional loader in the cli tests

• cli: add the functional loader in the cli tests

• functional: order entry points
• cli: add the functional loader in the cli tests

I created a new diff containing the evaluation branch: see D2807 instead.

• package.loader: add hook_branches method
• functional: create a branch named evaluation pointing to the evaluation commit

• functional: use the visit fixture in the test
• functional: add comments, licenses...
• functional: add provider and when in metadata.extrinsic
• functional: type _retrieve_sources

Thanks to @zimoun, https://guix.gnu.org/sources.json is now generated periodically (every hour). Each url is now a list.

@lewo : Let me know the new diff number. :-)

After some discussions with the SWH team, it is actually no longer the good way to fill the archive with our sources. Instead, I'm starting to write a loader which will be in charge of reading our sources.json and fill the archive. So, I'm closing this diff and will create a new diff with a loader in the next few days;)
There are also some advantages of implementing a loader: for instance, we could query the SWH API to know which sources of a specific sources.json file have been archived!

> @lewo: Does the version of the format should be bumped to 2 with this string-to-array modification?

While looking into this with @zimoun, we realized it would be nicer if url were an array of URLs (as is the case at https://guix.gnu.org/packages.json) rather than a single URL.

without changing now the crawler, i.e., the crawler can ingest only the first elem of the array and it will be modified later.

> If you are going to the FOSDEM, would be nice to meet you there to talk about next steps!

It would but i'm not going.

> If you are going to the FOSDEM, would be nice to meet you there to talk about next steps!

I'm already in Brussels and would be happy to meet!

A CI job is building a sources.json every day! The file is available at https://nix-community.github.io/nixpkgs-swh/sources.json ;)
This is a community CI (not hosted on main NixOS infrascture) which will allow me to iterate quickly on this file.

And fix another one :/

Fix the loader name in the test

Sorry for the delay... I will be more responsive now.

Rebase and change load-tar to load-archive-files

Fix ardumont comments

• Move sources.json mock to correct location

• Rename JSONLister to FunctionalPackageLister
• Fix test
• Cleaning

• wip: switch to the new format

LGTM, though I think we should define the git type right away. For that, we can probably reuse a format similar to that found at https://guix.gnu.org/packages.json, which looks like:

{
    "type": "git",
    "git_url": "https://github.com/pali/0xffff.git",
    "git_ref": "0.8"
}

... where git_ref can be a tag name or a commit ID.

WDYT?

I discussed with Ludo and we agreed on the fact the current packages.json file is not really suitable for the SWH usecase.
tl;dr the idea is to expose a list of sources instead of a list of packages.

Since Ludo (@civodul) posted this WIP feature on guix-devel mailing list [1] I am trying to follow this thread and I would like to help.

• [wip] swh.lister.json: Add lister getting sources from JSON file
• Rebased

By the way, i entertained the idea to write a guix lister with the current api, would that help (that could demo the tests to write as well)?

@ardumont @douardda Regarding the lister name, I agree the JSONLister name is not appropriate. @ardumont proposed FunctionalPackageManagerLister or FunctionalPackageLister but I'm not sure they are appropriated too :)

I think the hash and the version should be optional. I explain a bit why we can't always provide the version in D2025.

jsyk, @douardda and i are working on the package loader (T1389)

Using integrity attribute instead of hash. The format of this attribute follows the SRI specification.

This is a work in progress lister and we need to work on several points:

Hm, i don't know how to close this issue, but it should be closed now.
Thx.

Thx to this change (provided by @olasd) in the swh-docker-dev repository, tasks are exectued:

diff --git a/conf/loader.yml b/conf/loader.yml
index 4a4fb54..0cc07e6 100644
--- a/conf/loader.yml
+++ b/conf/loader.yml
@@ -5,6 +5,7 @@ storage:
 celery:
   task_broker: amqp://guest:guest@amqp//
   task_modules:
+    - swh.loader.package.tasks
     - swh.loader.debian.tasks
     - swh.loader.dir.tasks
     - swh.loader.git.tasks
@@ -16,6 +17,7 @@ celery:
     - swh.deposit.loader.tasks

Ok, thx a lot for your answers!