Page MenuHomeSoftware Heritage

Implement a Guix/Nix loader
Open, NormalPublic

Description

The objective of this task is to create a loader for Nix/Guix distributions. These distributions expose a sources.json containing all sources used at a specific moment.

This work could be split into the following diffs:

  • D2792: Create a loader able to load archives
  • D2807: Add a pointer in the snapshot to the commit used to generate the sources.json file
  • T2411: Deploy current nixguix loader version (fix issues along the way)
  • D2854: Support files that are not archive but plain text (stand-by)
  • Use all mirrors instead of just the first one.
  • Implement a mechanism to subscribe to a list of sources.json.
  • Support others types of source (git for instance)...

Event Timeline

ardumont triaged this task as Normal priority.Sep 9 2019, 10:08 AM
ardumont created this task.
ardumont renamed this task from Implement a Guix/GuixSD/Nix/NixOS lister to Implement a Guix/GuixSD/Nix/NixOS loader.Mar 5 2020, 7:28 PM
ardumont added a project: Core Loader.

Implementation of a lister started at D2025.

ardumont renamed this task from Implement a Guix/GuixSD/Nix/NixOS loader to Implement a Guix/Nix loader.Mar 5 2020, 7:30 PM
ardumont assigned this task to lewo.
lewo updated the task description. (Show Details)Mar 10 2020, 6:42 PM
lewo updated the task description. (Show Details)Mar 10 2020, 6:49 PM
lewo updated the task description. (Show Details)Mar 19 2020, 11:09 AM
ardumont updated the task description. (Show Details)Apr 2 2020, 4:34 PM
ardumont changed the status of subtask T2371: nixguix: fails to use previous visit snapshot from Open to Work in Progress.Apr 22 2020, 12:17 PM
ardumont added a comment.EditedMay 18 2020, 9:05 AM

For information, this has been running on staging daily for a while now without much issues. The sources.json are nix specifics for now.

@lewo decreased the number of potential errors [1] in the current loader by
removing potential issues in the sources.json [2]

Latest run on staging:

May 17 10:55:39 worker0 python3[1066]: [2020-05-17 10:55:39,333: INFO/ForkPoolWorker-3] Task swh.loader.package.nixguix.tasks.LoadNixguix[7982729e-96af-4700-8ae0-afa499a10830] succeeded in 2393.330544631928s: {'status': 'eventful', 'snapshot_id': 'a990925694c53453311030c652570901848b8a1d'}

[1] which will be improved upon later

[2] This is the files declaring the artifacts for the nixguix_loader to ingest.

ardumont changed the status of subtask T2411: Deploy nixguix loader from Open to Work in Progress.May 20 2020, 10:18 AM
ardumont updated the task description. (Show Details)May 26 2020, 12:04 PM
ardumont changed the status of subtask T2371: nixguix: fails to use previous visit snapshot from Open to Work in Progress.May 26 2020, 12:08 PM
ardumont updated the task description. (Show Details)May 26 2020, 5:27 PM