Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 8 2021

ardumont added a parent task for T3774: Deploy swh.loader.core v2.0.0: T2400: Ingest current and historical Ubuntu releases.
Dec 8 2021, 9:06 AM · System administration, Package Loader

Dec 7 2021

ardumont added a comment to T3774: Deploy swh.loader.core v2.0.0.

Schedule back origins with visit_type "deb" (ongoing)

Dec 7 2021, 6:56 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 6:54 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 6:54 PM · System administration, Package Loader
ardumont added a comment to T3774: Deploy swh.loader.core v2.0.0.

Same done in production.

Dec 7 2021, 6:12 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 6:06 PM · System administration, Package Loader
ardumont added a comment to T3774: Deploy swh.loader.core v2.0.0.

Triggered some debian loading in there.
And everything is working as expected.

Dec 7 2021, 5:49 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 5:14 PM · System administration, Package Loader
ardumont changed the status of T3774: Deploy swh.loader.core v2.0.0 from Open to Work in Progress.
Dec 7 2021, 4:13 PM · System administration, Package Loader
ardumont moved T3774: Deploy swh.loader.core v2.0.0 from Backlog to Weekly backlog on the System administration board.
Dec 7 2021, 4:13 PM · System administration, Package Loader
ardumont added a comment to T3774: Deploy swh.loader.core v2.0.0.

staging deployment done ^

Dec 7 2021, 4:13 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 4:08 PM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 3:02 PM · System administration, Package Loader
ardumont renamed T3774: Deploy swh.loader.core v2.0.0 from Deploy swh.loader.core v1.3.0 to Deploy swh.loader.core v2.0.0.
Dec 7 2021, 11:04 AM · System administration, Package Loader
ardumont added a project to T3774: Deploy swh.loader.core v2.0.0: System administration.
Dec 7 2021, 9:44 AM · System administration, Package Loader
ardumont updated the task description for T3774: Deploy swh.loader.core v2.0.0.
Dec 7 2021, 9:38 AM · System administration, Package Loader
ardumont triaged T3774: Deploy swh.loader.core v2.0.0 as Normal priority.
Dec 7 2021, 9:29 AM · System administration, Package Loader

Dec 6 2021

vlorentz closed T3743: Update the package-loader tutorial to reference releases instead of revisions as Resolved.
Dec 6 2021, 10:43 AM · Package Loader, Documentation
vlorentz added a revision to T3743: Update the package-loader tutorial to reference releases instead of revisions: D6729: package-loader-tutorial: Update to mention releases instead of revisions.
Dec 6 2021, 10:43 AM · Package Loader, Documentation

Dec 2 2021

vlorentz closed T3744: Add a checklist at the end of the package-loader tutorial as Resolved.
Dec 2 2021, 3:07 PM · Package Loader, Documentation

Dec 1 2021

vlorentz added a revision to T3744: Add a checklist at the end of the package-loader tutorial: D6710: package-loader-tutorial: Add a checklist.
Dec 1 2021, 2:24 PM · Package Loader, Documentation

Nov 26 2021

vlorentz added a comment to T3638: Make package loaders create releases objects instead of revisions.

Copy of an email I sent on 2021-11-17:

Nov 26 2021, 4:02 PM · Package Loader, Data Model, Archive content

Nov 22 2021

vlorentz closed T3636: Make the opam loader write extrinsic metadata, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Nov 22 2021, 2:44 PM · Package Loader, Data Model, Archive content
vlorentz closed T3638: Make package loaders create releases objects instead of revisions as Resolved.
Nov 22 2021, 2:43 PM · Package Loader, Data Model, Archive content
ardumont closed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Nov 22 2021, 2:29 PM · Package Loader, Data Model, Archive content
ardumont closed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 as Resolved.
Nov 22 2021, 2:29 PM · System administration, Package Loader, Data Model, Archive content
ardumont moved T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from deployed/landed/monitoring to Component upgrades on the System administration board.
Nov 22 2021, 2:29 PM · System administration, Package Loader, Data Model, Archive content
ardumont moved T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from in-progress to deployed/landed/monitoring on the System administration board.
Nov 22 2021, 2:05 PM · System administration, Package Loader, Data Model, Archive content
ardumont changed the status of T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, from Open to Work in Progress.
Nov 22 2021, 2:05 PM · Package Loader, Data Model, Archive content
ardumont changed the status of T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from Open to Work in Progress.
Nov 22 2021, 2:05 PM · System administration, Package Loader, Data Model, Archive content
ardumont added a project to T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3: System administration.
Nov 22 2021, 2:04 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 2:02 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 1:59 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 1:56 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 1:52 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 1:15 PM · System administration, Package Loader, Data Model, Archive content
ardumont renamed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from production: Deploy package loader v1.0, deposit server v0.16, lister v2.3 to production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Nov 22 2021, 1:06 PM · System administration, Package Loader, Data Model, Archive content
ardumont triaged T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 as Normal priority.
Nov 22 2021, 11:30 AM · System administration, Package Loader, Data Model, Archive content

Nov 20 2021

vlorentz triaged T3744: Add a checklist at the end of the package-loader tutorial as High priority.
Nov 20 2021, 1:50 PM · Package Loader, Documentation
vlorentz triaged T3743: Update the package-loader tutorial to reference releases instead of revisions as High priority.
Nov 20 2021, 1:49 PM · Package Loader, Documentation

Nov 10 2021

ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Nov 10 2021, 4:43 PM · Package Loader, Data Model, Archive content
ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Resolved.
Nov 10 2021, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont placed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 up for grabs.
Nov 10 2021, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from in-progress to deployed/landed/monitoring on the System administration board.
Nov 10 2021, 4:29 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont added a comment to T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.

At least loader deposit and npm [1] are fine.

Nov 10 2021, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Backlog to in-progress on the System administration board.
Nov 10 2021, 3:35 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont renamed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from staging: Deploy package loader v1.0 and deposit server v0.16 to staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, from Open to Work in Progress.
Nov 10 2021, 3:33 PM · Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Open to Work in Progress.
Nov 10 2021, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:32 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:21 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz added a parent task for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3: T3636: Make the opam loader write extrinsic metadata.
Nov 10 2021, 3:20 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont triaged T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Normal priority.
Nov 10 2021, 3:17 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content

Nov 9 2021

vlorentz added a revision to T3638: Make package loaders create releases objects instead of revisions: D6618: Document how each package loader populates fields..
Nov 9 2021, 12:39 PM · Package Loader, Data Model, Archive content

Nov 8 2021

vlorentz added revisions to T3638: Make package loaders create releases objects instead of revisions: D6616: Make package loaders write releases instead of revisions, D6617: Use release instead of revision as anchor in SWHID context instead..
Nov 8 2021, 11:58 AM · Package Loader, Data Model, Archive content
vlorentz added a comment to T3638: Make package loaders create releases objects instead of revisions.

Here is an overview of the fields (+ internal version name + branch name) used by each package loader, after D6616:

Nov 8 2021, 11:50 AM · Package Loader, Data Model, Archive content

Oct 8 2021

vlorentz added a subtask for T3638: Make package loaders create releases objects instead of revisions: T3636: Make the opam loader write extrinsic metadata.
Oct 8 2021, 2:32 PM · Package Loader, Data Model, Archive content
vlorentz added projects to T3638: Make package loaders create releases objects instead of revisions: Data Model, Package Loader.
Oct 8 2021, 2:30 PM · Package Loader, Data Model, Archive content

Sep 23 2021

vlorentz added a parent task for T3607: Document consistency guarantees of the loaders with respect to the storage: T3604: Document the architecture of all major packages/components.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation
vlorentz triaged T3607: Document consistency guarantees of the loaders with respect to the storage as Normal priority.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation

Aug 6 2021

ardumont updated the task description for T3469: package loader: Add missing filtering on redundant built revisions.
Aug 6 2021, 12:08 PM · Package Loader
ardumont triaged T3469: package loader: Add missing filtering on redundant built revisions as Normal priority.
Aug 6 2021, 12:07 PM · Package Loader

Jul 29 2021

vlorentz triaged T3449: pypi / package loader: handle dangling branches as Normal priority.
Jul 29 2021, 4:03 PM · Package Loader, Easy hack, PyPI loader

Apr 12 2021

vlorentz triaged T3244: Refactor the base package loader to load directly from tarballs as Low priority.
Apr 12 2021, 2:59 PM · Package Loader

Mar 23 2021

vlorentz closed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects, a subtask of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases, as Resolved.
Mar 23 2021, 2:33 PM · Package Loader, Storage manager, Extrinsic metadata

Dec 11 2020

ardumont renamed T2879: Finalize nixguix loader implementation from Improve nixguix loader implementations to Improve nixguix loader implementation.
Dec 11 2020, 3:49 PM · Nixguix loader, Package Loader
ardumont closed T1991: Implement a Guix/Nix loader as Resolved.
Dec 11 2020, 3:47 PM · Package Loader
ardumont added a comment to T1991: Implement a Guix/Nix loader.

The main loader is now deployed.
Another task got opened to keep the references on the possible improvments [1]

Dec 11 2020, 3:47 PM · Package Loader
ardumont updated the task description for T1991: Implement a Guix/Nix loader.
Dec 11 2020, 3:43 PM · Package Loader
ardumont triaged T2879: Finalize nixguix loader implementation as Normal priority.
Dec 11 2020, 3:42 PM · Nixguix loader, Package Loader

Nov 3 2020

ardumont moved T2368: deposit: Fix loader deposit inconsistency in revision committer date from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:06 PM · Package Loader, SWORD deposit

Nov 2 2020

vlorentz closed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Resolved.
Nov 2 2020, 12:23 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 23 2020

vlorentz added a revision to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4349: migrate_extrinsic_metadata: Write metadata on directories instead of revisions..
Oct 23 2020, 5:26 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added revisions to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4346: package loaders: write extrinsic metadata to directories instead of revisions., D4347: package loaders: write original_artifact metadata to directories instead of revisions..
Oct 23 2020, 5:01 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 14 2020

vlorentz removed a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 14 2020, 2:08 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects from Use intrinsic identifiers for RawExtrinsicMetadata objects to Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata
vlorentz triaged T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects as High priority.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata

Oct 12 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

FTR, olasd, douardda and I discussed an inconsistency in keys used in kafka, and decided to use hashes for all origin/visits/visit statuses; and doing the same for ext metadata in both kafka and the DB solves the issue about defining unicity.

Oct 12 2020, 1:52 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 12 2020, 1:06 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

@rdicosmo a full example of what?

Oct 12 2020, 10:57 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

The suggestion was to have extrinsic metadata on directories that come from a deposit of a bundle (e.g. .tar.gz or .zip file coming from HAL), instead of on a synthetic revision as is currently the case, so they can be accessed knowing the hash of the directory (which is an intrinsic id).

Oct 12 2020, 10:44 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 8 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

Alternatively, we could keep writing the metadata on revision/releases, and use the provenance service (when it's ready) to find them from a directory SWHID. What do you think?

Oct 8 2020, 11:47 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 6 2020

vlorentz updated the task description for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:45 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo updated subscribers of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:37 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases from Package loaders write extrinsic metadata on directories instead of revisions/releases to Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz triaged T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Normal priority.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata

Sep 15 2020

vsellier removed a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:25 AM · Package Loader
vsellier added a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:24 AM · Package Loader

Sep 2 2020

vlorentz triaged T2558: Loading tarballs with content permissions set to non-readable as Low priority.
Sep 2 2020, 10:53 PM · Package Loader

Aug 7 2020

ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont added a revision to T2510: nixguix: Filter out unsupported artifacts from ingestion: D3742: nixguix: Filter out unsupported artifact extensions.
Aug 7 2020, 5:44 PM · Package Loader

Aug 2 2020

lewo added a comment to T2510: nixguix: Filter out unsupported artifacts from ingestion.

I'm currently using the following regex to filter the exposed urls .tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$ but I'm pretty sure it could be improved.

Aug 2 2020, 9:27 PM · Package Loader

Jul 26 2020

ardumont updated the task description for T2510: nixguix: Filter out unsupported artifacts from ingestion.
Jul 26 2020, 6:24 AM · Package Loader
ardumont triaged T2510: nixguix: Filter out unsupported artifacts from ingestion as Normal priority.
Jul 26 2020, 6:23 AM · Package Loader

May 26 2020

ardumont updated the task description for T1991: Implement a Guix/Nix loader.
May 26 2020, 5:27 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2371: nixguix: fails to use previous visit snapshot as Resolved.

If that ever happens again (I don't expect it but I have been wrong in the
past), we will have a logged exception with some context.

May 26 2020, 2:44 PM · Package Loader