Page MenuHomeSoftware Heritage
Feed Advanced Search

Nov 20 2021

vlorentz triaged T3743: Update the package-loader tutorial to reference releases instead of revisions as High priority.
Nov 20 2021, 1:49 PM · Package Loader, Documentation

Nov 10 2021

ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Nov 10 2021, 4:43 PM · Package Loader, Data Model, Archive content
ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Resolved.
Nov 10 2021, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont placed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 up for grabs.
Nov 10 2021, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from in-progress to deployed/landed/monitoring on the System administration board.
Nov 10 2021, 4:29 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont added a comment to T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.

At least loader deposit and npm [1] are fine.

Nov 10 2021, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Backlog to in-progress on the System administration board.
Nov 10 2021, 3:35 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont renamed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from staging: Deploy package loader v1.0 and deposit server v0.16 to staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, from Open to Work in Progress.
Nov 10 2021, 3:33 PM · Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Open to Work in Progress.
Nov 10 2021, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:32 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Nov 10 2021, 3:21 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz added a parent task for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3: T3636: Make the opam loader write extrinsic metadata.
Nov 10 2021, 3:20 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont triaged T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Normal priority.
Nov 10 2021, 3:17 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content

Nov 9 2021

vlorentz added a revision to T3638: Make package loaders create releases objects instead of revisions: D6618: Document how each package loader populates fields..
Nov 9 2021, 12:39 PM · Package Loader, Data Model, Archive content

Nov 8 2021

vlorentz added revisions to T3638: Make package loaders create releases objects instead of revisions: D6616: Make package loaders write releases instead of revisions, D6617: Use release instead of revision as anchor in SWHID context instead..
Nov 8 2021, 11:58 AM · Package Loader, Data Model, Archive content
vlorentz added a comment to T3638: Make package loaders create releases objects instead of revisions.

Here is an overview of the fields (+ internal version name + branch name) used by each package loader, after D6616:

Nov 8 2021, 11:50 AM · Package Loader, Data Model, Archive content

Oct 8 2021

vlorentz added a subtask for T3638: Make package loaders create releases objects instead of revisions: T3636: Make the opam loader write extrinsic metadata.
Oct 8 2021, 2:32 PM · Package Loader, Data Model, Archive content
vlorentz added projects to T3638: Make package loaders create releases objects instead of revisions: Data Model, Package Loader.
Oct 8 2021, 2:30 PM · Package Loader, Data Model, Archive content

Sep 23 2021

vlorentz added a parent task for T3607: Document consistency guarantees of the loaders with respect to the storage: T3604: Document the architecture of all major packages/components.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation
vlorentz triaged T3607: Document consistency guarantees of the loaders with respect to the storage as Normal priority.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation

Aug 6 2021

ardumont updated the task description for T3469: package loader: Add missing filtering on redundant built revisions.
Aug 6 2021, 12:08 PM · Package Loader
ardumont triaged T3469: package loader: Add missing filtering on redundant built revisions as Normal priority.
Aug 6 2021, 12:07 PM · Package Loader

Jul 29 2021

vlorentz triaged T3449: pypi / package loader: handle dangling branches as Normal priority.
Jul 29 2021, 4:03 PM · Package Loader, Easy hack, PyPI loader

Apr 12 2021

vlorentz triaged T3244: Refactor the base package loader to load directly from tarballs as Low priority.
Apr 12 2021, 2:59 PM · Package Loader

Mar 23 2021

vlorentz closed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects, a subtask of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases, as Resolved.
Mar 23 2021, 2:33 PM · Package Loader, Storage manager, Extrinsic metadata

Dec 11 2020

ardumont renamed T2879: Finalize nixguix loader implementation from Improve nixguix loader implementations to Improve nixguix loader implementation.
Dec 11 2020, 3:49 PM · Nixguix loader, Package Loader
ardumont closed T1991: Implement a Guix/Nix loader as Resolved.
Dec 11 2020, 3:47 PM · Package Loader
ardumont added a comment to T1991: Implement a Guix/Nix loader.

The main loader is now deployed.
Another task got opened to keep the references on the possible improvments [1]

Dec 11 2020, 3:47 PM · Package Loader
ardumont updated the task description for T1991: Implement a Guix/Nix loader.
Dec 11 2020, 3:43 PM · Package Loader
ardumont triaged T2879: Finalize nixguix loader implementation as Normal priority.
Dec 11 2020, 3:42 PM · Nixguix loader, Package Loader

Nov 3 2020

ardumont moved T2368: deposit: Fix loader deposit inconsistency in revision committer date from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:06 PM · Package Loader, SWORD deposit

Nov 2 2020

vlorentz closed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Resolved.
Nov 2 2020, 12:23 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 23 2020

vlorentz added a revision to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4349: migrate_extrinsic_metadata: Write metadata on directories instead of revisions..
Oct 23 2020, 5:26 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added revisions to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: D4346: package loaders: write extrinsic metadata to directories instead of revisions., D4347: package loaders: write original_artifact metadata to directories instead of revisions..
Oct 23 2020, 5:01 PM · Package Loader, Storage manager, Extrinsic metadata

Oct 14 2020

vlorentz removed a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 14 2020, 2:08 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects from Use intrinsic identifiers for RawExtrinsicMetadata objects to Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata
vlorentz triaged T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects as High priority.
Oct 14 2020, 2:01 PM · Data Model, Storage manager, Extrinsic metadata

Oct 12 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

FTR, olasd, douardda and I discussed an inconsistency in keys used in kafka, and decided to use hashes for all origin/visits/visit statuses; and doing the same for ext metadata in both kafka and the DB solves the issue about defining unicity.

Oct 12 2020, 1:52 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a subtask for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases: T2686: Use hashes for all kafka keys.
Oct 12 2020, 1:06 PM · Package Loader, Storage manager, Extrinsic metadata
vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

@rdicosmo a full example of what?

Oct 12 2020, 10:57 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

The suggestion was to have extrinsic metadata on directories that come from a deposit of a bundle (e.g. .tar.gz or .zip file coming from HAL), instead of on a synthetic revision as is currently the case, so they can be accessed knowing the hash of the directory (which is an intrinsic id).

Oct 12 2020, 10:44 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 8 2020

vlorentz added a comment to T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.

Alternatively, we could keep writing the metadata on revision/releases, and use the provenance service (when it's ready) to find them from a directory SWHID. What do you think?

Oct 8 2020, 11:47 AM · Package Loader, Storage manager, Extrinsic metadata

Oct 6 2020

vlorentz updated the task description for T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:45 AM · Package Loader, Storage manager, Extrinsic metadata
rdicosmo updated subscribers of T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:37 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz renamed T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases from Package loaders write extrinsic metadata on directories instead of revisions/releases to Package loaders should write extrinsic metadata on directories instead of revisions/releases.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata
vlorentz triaged T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases as Normal priority.
Oct 6 2020, 10:30 AM · Package Loader, Storage manager, Extrinsic metadata

Sep 15 2020

vsellier removed a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:25 AM · Package Loader
vsellier added a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
Sep 15 2020, 11:24 AM · Package Loader

Sep 2 2020

vlorentz triaged T2558: Loading tarballs with content permissions set to non-readable as Low priority.
Sep 2 2020, 10:53 PM · Package Loader

Aug 7 2020

ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont closed T2510: nixguix: Filter out unsupported artifacts from ingestion, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
Aug 7 2020, 11:51 PM · Package Loader
ardumont added a revision to T2510: nixguix: Filter out unsupported artifacts from ingestion: D3742: nixguix: Filter out unsupported artifact extensions.
Aug 7 2020, 5:44 PM · Package Loader

Aug 2 2020

lewo added a comment to T2510: nixguix: Filter out unsupported artifacts from ingestion.

I'm currently using the following regex to filter the exposed urls .tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$ but I'm pretty sure it could be improved.

Aug 2 2020, 9:27 PM · Package Loader

Jul 26 2020

ardumont updated the task description for T2510: nixguix: Filter out unsupported artifacts from ingestion.
Jul 26 2020, 6:24 AM · Package Loader
ardumont triaged T2510: nixguix: Filter out unsupported artifacts from ingestion as Normal priority.
Jul 26 2020, 6:23 AM · Package Loader

May 26 2020

ardumont updated the task description for T1991: Implement a Guix/Nix loader.
May 26 2020, 5:27 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2411: Deploy nixguix loader, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
May 26 2020, 2:53 PM · Package Loader
ardumont closed T2371: nixguix: fails to use previous visit snapshot as Resolved.

If that ever happens again (I don't expect it but I have been wrong in the
past), we will have a logged exception with some context.

May 26 2020, 2:44 PM · Package Loader
ardumont closed T2371: nixguix: fails to use previous visit snapshot, a subtask of T1991: Implement a Guix/Nix loader, as Resolved.
May 26 2020, 2:44 PM · Package Loader
ardumont closed D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 2:14 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 2:07 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 2:05 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.

rework commit message again (last time!)

May 26 2020, 2:04 PM · Package Loader
ardumont updated the summary of D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 2:04 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.
  • Adapt according to review (log.exception, prefer unexpected to divergent)
  • Rework commit message to align with the diff
May 26 2020, 2:03 PM · Package Loader
vlorentz accepted D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 1:00 PM · Package Loader
ardumont retitled D2949: nixguix: catch and log artifact resolution failures from nixguix: catch eventual artifact resolution failures and logs those to nixguix: catch and log artifact resolution failures.
May 26 2020, 12:11 PM · Package Loader
ardumont changed the status of T2371: nixguix: fails to use previous visit snapshot, a subtask of T1991: Implement a Guix/Nix loader, from Open to Work in Progress.
May 26 2020, 12:08 PM · Package Loader
ardumont changed the status of T2371: nixguix: fails to use previous visit snapshot from Open to Work in Progress.

D2949 updated accordingly

May 26 2020, 12:08 PM · Package Loader
ardumont claimed T2411: Deploy nixguix loader.
May 26 2020, 12:08 PM · Package Loader
ardumont claimed T2371: nixguix: fails to use previous visit snapshot.
May 26 2020, 12:07 PM · Package Loader
ardumont updated the task description for T1991: Implement a Guix/Nix loader.
May 26 2020, 12:04 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 11:59 AM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.

Drop unnecessary log instructions

May 26 2020, 11:57 AM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 26 2020, 11:57 AM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.
  • Fix existing source.json which should not have been modified in the first place
  • Add test scenario around the modification the diff introduces
May 26 2020, 11:55 AM · Package Loader
ardumont added inline comments to D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 10:15 AM · Package Loader
ardumont planned changes to D2949: nixguix: catch and log artifact resolution failures.
May 26 2020, 10:15 AM · Package Loader
ardumont retitled D2949: nixguix: catch and log artifact resolution failures from nixguix: Fix "integrity" field resolution to nixguix: catch eventual artifact resolution failures and logs those.
May 26 2020, 10:05 AM · Package Loader
Harbormaster failed remote builds in B12491: Diff 11299 for D2949: nixguix: catch and log artifact resolution failures!
May 26 2020, 10:04 AM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build has FAILED

May 26 2020, 10:04 AM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.

Rebase on latest master

May 26 2020, 10:02 AM · Package Loader
ardumont added a comment to T2371: nixguix: fails to use previous visit snapshot.

It's not invalid in the end, we hit it in production now.

May 26 2020, 9:46 AM · Package Loader

May 25 2020

ardumont added a comment to T2411: Deploy nixguix loader.

and done:

May 25 17:17:10 worker01 python3[603377]: [2020-05-25 17:17:10,297: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[5be5167d-3783-4fc9-b18e-116424949d0a] succeeded in 3153.633277214016s: {'status': 'eventful', 'snapshot_id': '76f38fb09efe9f10305c50f5cf083b008217256f'}
May 25 2020, 8:18 PM · Package Loader
ardumont added a comment to T2411: Deploy nixguix loader.

Even simpler than D2949, filtering out the culprit branch "evaluation" (targetting nixpkgs) and we are back on track.

May 25 2020, 6:54 PM · Package Loader
ardumont abandoned D2949: nixguix: catch and log artifact resolution failures.

Replaced by D3178

May 25 2020, 6:52 PM · Package Loader
ardumont retitled D2949: nixguix: catch and log artifact resolution failures from nixguix: Fix "integrity" field resolution out of a snapshot to nixguix: Fix "integrity" field resolution.
May 25 2020, 6:50 PM · Package Loader
ardumont added a revision to T2411: Deploy nixguix loader: D3178: nixguix: Override known_artifacts to filter out evaluation branch.
May 25 2020, 6:44 PM · Package Loader
Harbormaster failed remote builds in B12486: Diff 11295 for D2949: nixguix: catch and log artifact resolution failures!
May 25 2020, 6:38 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build has FAILED

May 25 2020, 6:38 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.
  • nixguix: Add a test around the wrong metadata revision structure detection
May 25 2020, 6:36 PM · Package Loader
ardumont added inline comments to D2949: nixguix: catch and log artifact resolution failures.
May 25 2020, 6:26 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 25 2020, 5:49 PM · Package Loader
ardumont updated the diff for D2949: nixguix: catch and log artifact resolution failures.

Rework according to review
(still working on a test)

May 25 2020, 5:48 PM · Package Loader
lewo added inline comments to D2949: nixguix: catch and log artifact resolution failures.
May 25 2020, 5:33 PM · Package Loader
ardumont added a comment to D2949: nixguix: catch and log artifact resolution failures.

Working on adding one test which pass in there (load from archive loader, then load from nixguix whose listing list the same artifact...)

I actually don't hit the issue in a test...

Well, i did through patching...

May 25 2020, 5:33 PM · Package Loader
swh-public-ci added a comment to D2949: nixguix: catch and log artifact resolution failures.

Build is green

May 25 2020, 4:13 PM · Package Loader