Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 8 2023

gitlab-migration closed T4609: nipxkgs manifests list "git" origins as "urls", a subtask of T3781: Replace the Nixguix loader with a lister, as Migrated.
Jan 8 2023, 10:25 PM · Data Model, Nixguix loader
gitlab-migration closed T4609: nipxkgs manifests list "git" origins as "urls" as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:25 PM · Data Model, Nixguix loader
gitlab-migration closed T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:25 PM · Data Model, Nixguix loader
gitlab-migration closed T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes, a subtask of T3781: Replace the Nixguix loader with a lister, as Migrated.
Jan 8 2023, 10:25 PM · Data Model, Nixguix loader
gitlab-migration closed T4538: Consider archiving NAR hashes as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:24 PM · SVN Loader, Tarball loader, Nixguix loader
gitlab-migration closed T3781: Replace the Nixguix loader with a lister, a subtask of T2879: Finalize nixguix loader implementation, as Migrated.
Jan 8 2023, 10:23 PM · Nixguix loader, Package Loader
gitlab-migration closed T3781: Replace the Nixguix loader with a lister as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Data Model, Nixguix loader
gitlab-migration changed the status of T4662: staging: Deploy nixguix lister and loader, a subtask of T3781: Replace the Nixguix loader with a lister, from Invalid to Migrated.
Jan 8 2023, 10:04 PM · Data Model, Nixguix loader
gitlab-migration changed the status of T4662: staging: Deploy nixguix lister and loader from Invalid to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:04 PM · Data Model, Nixguix loader
gitlab-migration changed the status of T3763: Unstuck nixguix loading visits from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:35 PM · Nixguix loader
gitlab-migration changed the status of T3294: nixguix: Add support for pseudo-URLs with a missing schema, a subtask of T3781: Replace the Nixguix loader with a lister, from Resolved to Migrated.
Jan 8 2023, 4:34 PM · Data Model, Nixguix loader
gitlab-migration changed the status of T3294: nixguix: Add support for pseudo-URLs with a missing schema from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:34 PM · Easy hack, Nixguix loader
gitlab-migration changed the status of T2879: Finalize nixguix loader implementation from Wontfix to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:32 PM · Nixguix loader, Package Loader
gitlab-migration changed the status of T2687: nixguix: Add support for downloads over FTP from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:31 PM · Nixguix loader, Easy hack
gitlab-migration changed the status of T2687: nixguix: Add support for downloads over FTP, a subtask of T2879: Finalize nixguix loader implementation, from Resolved to Migrated.
Jan 8 2023, 4:31 PM · Nixguix loader, Package Loader

Nov 3 2022

bchauvet added a parent task for T4662: staging: Deploy nixguix lister and loader: T4079: Extend archive coverage.
Nov 3 2022, 10:17 AM · Data Model, Nixguix loader

Oct 31 2022

ardumont added a comment to T4538: Consider archiving NAR hashes.

Oh awesome! Thanks for pushing forward. :-)

Oct 31 2022, 12:14 PM · SVN Loader, Tarball loader, Nixguix loader

Oct 26 2022

ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 26 2022, 5:48 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

With latest diffs, the filtering seems to sort properly the files and tarballs for the guix manifest:

Oct 26 2022, 2:44 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 26 2022, 2:34 PM · Data Model, Nixguix loader
ardumont closed T4662: staging: Deploy nixguix lister and loader, a subtask of T3781: Replace the Nixguix loader with a lister, as Invalid.
Oct 26 2022, 2:33 PM · Data Model, Nixguix loader
ardumont closed T4662: staging: Deploy nixguix lister and loader as Invalid.

/me sighs

Oct 26 2022, 2:33 PM · Data Model, Nixguix loader
ardumont triaged T4662: staging: Deploy nixguix lister and loader as Normal priority.
Oct 26 2022, 2:31 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 26 2022, 2:29 PM · Data Model, Nixguix loader

Oct 25 2022

ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 25 2022, 6:57 PM · Data Model, Nixguix loader
ardumont updated subscribers of T3781: Replace the Nixguix loader with a lister.

Last analysis without [1]. That last diff should fix the key entries marked with the key 'only-version-should-be-tarball'.

Oct 25 2022, 6:53 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8774: nixguix: Use content-disposition from http head request if provided.
Oct 25 2022, 5:50 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8773: nixguix: Deal with edge case url with version instead of extension.
Oct 25 2022, 5:29 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 25 2022, 2:50 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8763: nixguix: Allow lister to ignore specific extensions.
Oct 25 2022, 10:41 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8761: nixguix/test: Add all supported tarball extensions to test manifest.
Oct 25 2022, 10:09 AM · Data Model, Nixguix loader

Oct 24 2022

ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8758: swh.core.tarball: Wire support to existing tarball.
Oct 24 2022, 4:38 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Checks that newly detected extensions are actually supported already.
Summary [1] and the actual checks [2]:

Oct 24 2022, 4:36 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8757: Add support for more tarball/zip extensions.
Oct 24 2022, 3:53 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

I had a pass on extensions to further check what's a tarball or not [1]

Oct 24 2022, 3:19 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 24 2022, 3:17 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Improved version with noisy urls printed alongside the hash output [1] [2]:

Oct 24 2022, 3:16 PM · Data Model, Nixguix loader

Oct 22 2022

zimoun added a comment to T4538: Consider archiving NAR hashes.

Oh awesome! Thanks for pushing forward. :-)

Oct 22 2022, 12:46 PM · SVN Loader, Tarball loader, Nixguix loader

Oct 21 2022

ardumont updated subscribers of T4538: Consider archiving NAR hashes.

fwiw, I've iterated a bit over @zimoun's code and pushed it into the snippets repository
(see commits above and their commit description message). It's also able to deal with
git, hg and svn trees (ignoring their respective top metadata folder .git, .svn, ...
without impacting the performance).

Oct 21 2022, 3:46 PM · SVN Loader, Tarball loader, Nixguix loader

Oct 19 2022

gitlab-migration changed the status of T3791: Deploy swh.loader.core v2.1 from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:05 PM · System administration, Nixguix loader, Package Loader

Oct 14 2022

zimoun added a comment to T4538: Consider archiving NAR hashes.
import os, stat, hashlib, io
from pathlib import Path
Oct 14 2022, 10:23 AM · SVN Loader, Tarball loader, Nixguix loader
zimoun added a comment to T4538: Consider archiving NAR hashes.

This tree

$ tree foo
foo
├── bar
│   └── exe
└── baz
Oct 14 2022, 10:23 AM · SVN Loader, Tarball loader, Nixguix loader

Oct 11 2022

zimoun added a comment to T4538: Consider archiving NAR hashes.

To be complete, the integrity checksum that Guix provides (here one example about the package guix-modules),

Oct 11 2022, 8:19 PM · SVN Loader, Tarball loader, Nixguix loader
zimoun added a comment to T4538: Consider archiving NAR hashes.

Related to P1473.

Oct 11 2022, 7:59 PM · SVN Loader, Tarball loader, Nixguix loader

Oct 7 2022

ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

With D8637, listing is less noisy [1] (code [2]):

Oct 7 2022, 2:22 PM · Data Model, Nixguix loader

Oct 6 2022

ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 6:15 PM · Data Model, Nixguix loader
ardumont closed T3294: nixguix: Add support for pseudo-URLs with a missing schema as Resolved.

The code is landed. It will be released at the same time as the lister/loader stack.
So let's close this.

Oct 6 2022, 6:13 PM · Easy hack, Nixguix loader
ardumont closed T3294: nixguix: Add support for pseudo-URLs with a missing schema, a subtask of T3781: Replace the Nixguix loader with a lister, as Resolved.
Oct 6 2022, 6:13 PM · Data Model, Nixguix loader
ardumont updated the task description for T4609: nipxkgs manifests list "git" origins as "urls".
Oct 6 2022, 6:12 PM · Data Model, Nixguix loader
ardumont triaged T4609: nipxkgs manifests list "git" origins as "urls" as Normal priority.
Oct 6 2022, 6:11 PM · Data Model, Nixguix loader
ardumont updated the task description for T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes.
Oct 6 2022, 6:08 PM · Data Model, Nixguix loader
ardumont added a comment to T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes.

I tried to have a look at the code and try to run it but fail [1].

Oct 6 2022, 4:54 PM · Data Model, Nixguix loader
ardumont updated the task description for T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes.
Oct 6 2022, 4:02 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8637: nixguix: Exclude faulty "recursive" file origins from listing.
Oct 6 2022, 4:02 PM · Data Model, Nixguix loader
ardumont added a revision to T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes: D8637: nixguix: Exclude faulty "recursive" file origins from listing.
Oct 6 2022, 4:02 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 3:36 PM · Data Model, Nixguix loader
ardumont triaged T4608: nixpkgs manifests list "recursive" file which are missing information to recompute their hashes as Normal priority.
Oct 6 2022, 3:32 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 2:44 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

The actual nixpkgs manifests are either not built properly or not complete yet. They
sometimes are referencing hash we cannot compute back as only the derivation is
containing the information [1] [2].

Oct 6 2022, 2:37 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 2:31 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 2:08 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8636: ContentLoader: Allow nar computation checks.
Oct 6 2022, 1:03 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 10:53 AM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Finally, more concentrated frequency dict:

Oct 6 2022, 9:17 AM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

It must be more interesting to read it with a frequency [1]:

Oct 6 2022, 9:10 AM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Out of the paste [1] (csv extract from swh-scheduler dev db after 3 lister runs on
docker), here is the state of detected files [2] so far (computed with [3]):

Oct 6 2022, 9:04 AM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 6 2022, 8:50 AM · Data Model, Nixguix loader

Oct 5 2022

ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 4:57 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8632: nixguix: Refactor by renaming success or failure the different datasets.
Oct 5 2022, 4:51 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8631: nixguix: Deal with manifest entries without an integrity field.
Oct 5 2022, 4:01 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 3:06 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 3:00 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 2:53 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8630: {Cnt|Dir}Loader: Fix standard/nar hash mismatch behavior to fail loading.
Oct 5 2022, 2:48 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 12:04 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8627: base-buster/Dockerfile: Install nix binaries in buster image.
Oct 5 2022, 11:57 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8626: nixguix: Improve is_tarball detection pattern.
Oct 5 2022, 11:53 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8625: docker: Install nix binaries in swh/stack image.
Oct 5 2022, 11:30 AM · Data Model, Nixguix loader
ardumont added a revision to T4538: Consider archiving NAR hashes: D8625: docker: Install nix binaries in swh/stack image.
Oct 5 2022, 11:30 AM · SVN Loader, Tarball loader, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 11:25 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8624: nixguix: Improve further tarball detection.
Oct 5 2022, 11:12 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8623: core.tarball: Add missing mimetype to the list.
Oct 5 2022, 11:09 AM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 10:34 AM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8621: wip: Add nixguix lister and loader.
Oct 5 2022, 10:34 AM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 10:11 AM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 5 2022, 10:06 AM · Data Model, Nixguix loader

Oct 4 2022

ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8620: nixguix: Improve git origins detection.
Oct 4 2022, 11:32 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8619: nixguix: Improve tarball detection.
Oct 4 2022, 8:44 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8618: DirectoryLoader: Check nar hashes when provided.
Oct 4 2022, 7:33 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 4 2022, 4:53 PM · Data Model, Nixguix loader
ardumont added a revision to T3294: nixguix: Add support for pseudo-URLs with a missing schema: D8614: nixguix: Add support for listing origins with "recursive" integrity.
Oct 4 2022, 4:50 PM · Easy hack, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8614: nixguix: Add support for listing origins with "recursive" integrity.
Oct 4 2022, 4:50 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Another data point about ^. It's not important for the guix manifest [1]. We can keep a
compatible behavior for it and slightly improve the listing behavior for nixpkgs as it's
important for those [2].

Oct 4 2022, 4:33 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 4 2022, 4:17 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

With the gazillion of new diffs on top of the origin lister code, we can now also list
the nixpkgs-unstable-full.json manifests [1]

Oct 4 2022, 4:17 PM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Oct 4 2022, 4:05 PM · Data Model, Nixguix loader
ardumont added a revision to T3294: nixguix: Add support for pseudo-URLs with a missing schema: D8612: nixguix: Add support for pseudo url with missing schema.
Oct 4 2022, 4:01 PM · Easy hack, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8612: nixguix: Add support for pseudo url with missing schema.
Oct 4 2022, 4:01 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8611: nixguix: Deal with connection error with server.
Oct 4 2022, 2:57 PM · Data Model, Nixguix loader
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8610: nixguix: Refactor by cleaning up unneeded code.
Oct 4 2022, 2:40 PM · Data Model, Nixguix loader