Another one bites the dust [1]

For the content loader i have mostly checksums mismatches [1].
It seems the integrity from the manifest is either wrong (or some in-place update occurred in the respective servers [2])

Run through docker for directory:

docker run on the lister:

17:36:23 swh-scheduler@localhost:5433=# select now(), visit_type, lister_id, count(*) from listed_origins where lister_id = ( select id from listers where name='nixguix' and instance_name='nix-community.github.io') group by visit_type, lister_id;
+-------------------------------+------------+--------------------------------------+-------+
|              now              | visit_type |              lister_id               | count |
+-------------------------------+------------+--------------------------------------+-------+
| 2022-10-03 15:44:20.179895+00 | git        | 3f5c040a-6247-4ef3-a812-36f4b9ceafeb |     1 |
| 2022-10-03 15:44:20.179895+00 | file       | 3f5c040a-6247-4ef3-a812-36f4b9ceafeb |    87 |
| 2022-10-03 15:44:20.179895+00 | tar        | 3f5c040a-6247-4ef3-a812-36f4b9ceafeb | 31130 |
+-------------------------------+------------+--------------------------------------+-------+
(3 rows)

Thanks for raising the questions.

Hum, for the 7 false, I have to check. For the 88 packages with no-origin, it is more
annoying. Well, some are metapackages as gcc-toolchain, so they can be skipped. Is it
ok for you to let this 'no-origin' type? For some others, I have to check if they are
covered elsewhere.

For ^, something like this would do [1]

Thanks for all that ^! And great pointers!

• artifacts url which are mostly tarballs [1] and sometimes files [2]
• dvcs repositories delegated to dedicated loader to ingestion: svn [3], hg [4], git [5] (out of guix manifest)
• Other stuff can be ignored as we don't have anything relevant to ingest [6]

I've moved around the parent task as the lister should be able to do something about this.

This perimeter enters the scope of T3781.

yes, nix too. As far as i remember, it's NAR for Nix ARchive ;)

Some more information regarding extensions supported in nixpkgs and guix manifests:

In [33]: sources = "https://nix-community.github.io/nixpkgs-swh/sources-unstable.json"

So taking a bit more look into this possible new lister, we'd end up with the following
possible outputs:

• artifacts url which are mostly tarballs [1] and sometimes files [2]
• dvcs repositories delegated to dedicated loader to ingestion: svn [3], hg [4], git [5] (out of guix manifest)
• Other stuff can be ignored as we don't have anything relevant to ingest [6]

Another argument: currently, there is always at least some failures when loading real
Nix and Guix repositories, so visits always have status partial; which prevents them
from being listed in
https://archive.softwareheritage.org/browse/search/?q=&with_visit=true&with_content=true&visit_type=nixguix
(but we get results when un-checking " only show origins visited at least once")

Another argument: currently, there is always at least some failures when loading real Nix and Guix repositories, so visits always have status partial; which prevents them from being listed in https://archive.softwareheritage.org/browse/search/?q=&with_visit=true&with_content=true&visit_type=nixguix (but we get results when un-checking " only show origins visited at least once")

I'm growing fond of this idea.
That should take less time to refactor it now that we improved the lister scaffolding and
that we mostly know what the perimeter of the nixguix loader is.

That may be superseded by the task T3781 which proposes to rewrite that loader as a lister.
And then delegate the origins listed by it to loaders (git, archive, etc...)

Nixguix loader is ok.

Maybe another data point for the discussion is that the nixguix loader currently only
shows 1 origin for guix and 1 for nixos [well nixpkgs really} in the coverage part [1].
Which is somewhat true... but... feels weird at the same time.

Now the loader nixguix fails with [1]

Not saying no.

It finished during the week end [1].
As a secondary check, the scheduling is not stuck [2].

Deployed and triggered another scheduling.

v1.2.1 in the oven.

Another somewhat related issue [1]
Incoming diff to fix it.

Deployed with loader.core v1.2.
I've just scheduled a new nixguix load run for both guix and nixpkgs.
Let's see if that unblocks it.

Ah yes, indeed. Make release_extid_targets a set to deduplicate target SWHIDs, and the assertion should be fine.

@vlorentz, is that failing assertion still True? (since we moved from synthetic revision to release)