Page MenuHomeSoftware Heritage

Package LoaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Recent Activity

Mon, Apr 12

vlorentz triaged T3244: Refactor the base package loader to load directly from tarballs as Low priority.
Mon, Apr 12, 2:59 PM · Package Loader

Mar 23 2021

vlorentz closed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects, a subtask of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases, as Resolved.
Mar 23 2021, 2:33 PM · Package Loader, Storage manager, Extrinsic metadata

Dec 11 2020

ardumont renamed T2879: Improve nixguix loader implementation from Improve nixguix loader implementations to Improve nixguix loader implementation.
Dec 11 2020, 3:49 PM · Nixguix loader, Package Loader
ardumont closed T1991: Implement a Guix/Nix loader as Resolved.
Dec 11 2020, 3:47 PM · Package Loader
ardumont added a comment to T1991: Implement a Guix/Nix loader.

The main loader is now deployed.
Another task got opened to keep the references on the possible improvments [1]

Dec 11 2020, 3:47 PM · Package Loader
ardumont updated the task description for T1991: Implement a Guix/Nix loader.
Dec 11 2020, 3:43 PM · Package Loader
ardumont triaged T2879: Improve nixguix loader implementation as Normal priority.
Dec 11 2020, 3:42 PM · Nixguix loader, Package Loader

Nov 3 2020

ardumont moved T2368: deposit: Fix loader deposit inconsistency in revision committer date from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:06 PM · Package Loader, SWORD deposit

Nov 2 2020

vlorentz closed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Resolved.
Nov 2 2020, 12:23 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 23 2020

vlorentz added a revision to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4349: migrate_extrinsic_metadata: Write metadata on directories instead of revisions..
Oct 23 2020, 5:26 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added revisions to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4346: package loaders: write extrinsic metadata to directories instead of revisions., D4347: package loaders: write original_artifact metadata to directories instead of revisions..
Oct 23 2020, 5:01 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 14 2020

vlorentz removed a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 14 2020, 2:08 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects from Use intrinsic identifiers for RawExtrinsicMetadata objects to Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata
vlorentz triaged T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects as High priority.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata

Oct 12 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

FTR, olasd, douardda and I discussed an inconsistency in keys used in kafka, and decided to use hashes for all origin/visits/visit statuses; and doing the same for ext metadata in both kafka and the DB solves the issue about defining unicity.

Oct 12 2020, 1:52 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 12 2020, 1:06 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

@rdicosmo a full example of what?

Oct 12 2020, 10:57 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

The suggestion was to have extrinsic metadata on directories that come from a deposit of a bundle (e.g. .tar.gz or .zip file coming from HAL), instead of on a synthetic revision as is currently the case, so they can be accessed knowing the hash of the directory (which is an intrinsic id).

Oct 12 2020, 10:44 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 8 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

Alternatively, we could keep writing the metadata on revision/releases, and use the provenance service (when it's ready) to find them from a directory SWHID. What do you think?

Oct 8 2020, 11:47 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 6 2020

vlorentz updated the task description for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:45 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo updated subscribers of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:37 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases from Package loaders write extrinsic metadata on directories instead of revisions/releases to Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz triaged T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Normal priority.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata

Sep 15 2020

vsellier removed a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:25 AM · Package Loader
vsellier added a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:24 AM · Package Loader

Sep 2 2020

vlorentz triaged T2558: Loading tarballs with content permissions set to non-readable as Low priority.
Sep 2 2020, 10:53 PM · Package Loader

Aug 7 2020

ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont added a revision to T2510: nixguix: Filter out unsupported artifacts from ingestion: D3742: nixguix: Filter out unsupported artifact extensions.
Aug 7 2020, 5:44 PM · Package Loader

Aug 2 2020

lewo added a comment to T2510: nixguix: Filter out unsupported artifacts from ingestion.

I'm currently using the following regex to filter the exposed urls .tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$ but I'm pretty sure it could be improved.

Aug 2 2020, 9:27 PM · Package Loader

Jul 26 2020

ardumont updated the task description for T2510: nixguix: Filter out unsupported artifacts from ingestion.
Jul 26 2020, 6:24 AM · Package Loader
ardumont triaged T2510: nixguix: Filter out unsupported artifacts from ingestion as Normal priority.
Jul 26 2020, 6:23 AM · Package Loader

May 26 2020

ardumont updated the task description for T1991: Implement a Guix/Nix loader.
May 26 2020, 5:27 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2371: nixguix: fails to use previous visit snapshot as Resolved.

If that ever happens again (I don't expect it but I have been wrong in the
past), we will have a logged exception with some context.

May 26 2020, 2:44 PM · Package Loader
ardumont closed T2371: nixguix: fails to use previous visit snapshot, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
May 26 2020, 2:44 PM · Package Loader
ardumont closed D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 2:14 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 2:07 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 2:05 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.

rework commit message again (last time!)

May 26 2020, 2:04 PM · Package Loader
ardumont updated the summary of D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 2:04 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.
  • Adapt according to review (log.exception, prefer unexpected to divergent)
  • Rework commit message to align with the diff
May 26 2020, 2:03 PM · Package Loader
vlorentz accepted D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 1:00 PM · Package Loader
ardumont retitled D2949: nixguix: catch and log artifact resolution failures from nixguix: catch eventual artifact resolution failures and logs those to nixguix: catch and log artifact resolution failures.
May 26 2020, 12:11 PM · Package Loader
ardumont changed the status of T2371: nixguix: fails to use previous visit snapshot, a subtask of T1991: Implement a Guix/Nix loader, from Open to Work in Progress.
May 26 2020, 12:08 PM · Package Loader
ardumont changed the status of T2371: nixguix: fails to use previous visit snapshot from Open to Work in Progress.

D2949 updated accordingly

May 26 2020, 12:08 PM · Package Loader
ardumont claimed T2411: Deploy nixguix loader.
May 26 2020, 12:08 PM · Package Loader
ardumont claimed T2371: nixguix: fails to use previous visit snapshot.
May 26 2020, 12:07 PM · Package Loader
ardumont updated the task description for T1991: Implement a Guix/Nix loader.
May 26 2020, 12:04 PM · Package Loader