Page MenuHomeSoftware Heritage

Archive contentFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

stuff related to content (of all kinds, not only "blobs") that is already stored in the Software Heritage archive

Recent Activity

Fri, Nov 26

vlorentz added a comment to T3638: Make package loaders create releases objects instead of revisions.

Copy of an email I sent on 2021-11-17:

Fri, Nov 26, 4:02 PM · Package Loader, Data Model, Archive content

Mon, Nov 22

vlorentz closed T3636: Make the opam loader write extrinsic metadata, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Mon, Nov 22, 2:44 PM · Package Loader, Data Model, Archive content
vlorentz closed T3638: Make package loaders create releases objects instead of revisions, a subtask of T1258: Synthesize release objects for all upstream things that match the concept of a release, as Resolved.
Mon, Nov 22, 2:44 PM · Archive content
vlorentz closed T3638: Make package loaders create releases objects instead of revisions as Resolved.
Mon, Nov 22, 2:43 PM · Package Loader, Data Model, Archive content
ardumont closed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Mon, Nov 22, 2:29 PM · Package Loader, Data Model, Archive content
ardumont closed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 as Resolved.
Mon, Nov 22, 2:29 PM · System administration, Package Loader, Data Model, Archive content
ardumont moved T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from deployed/landed to Component upgrades on the System administration board.
Mon, Nov 22, 2:29 PM · System administration, Package Loader, Data Model, Archive content
ardumont moved T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from in-progress to deployed/landed on the System administration board.
Mon, Nov 22, 2:05 PM · System administration, Package Loader, Data Model, Archive content
ardumont changed the status of T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, from Open to Work in Progress.
Mon, Nov 22, 2:05 PM · Package Loader, Data Model, Archive content
ardumont changed the status of T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from Open to Work in Progress.
Mon, Nov 22, 2:05 PM · System administration, Package Loader, Data Model, Archive content
ardumont added a project to T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3: System administration.
Mon, Nov 22, 2:04 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 2:02 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 1:59 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 1:56 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 1:52 PM · System administration, Package Loader, Data Model, Archive content
ardumont updated the task description for T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 1:15 PM · System administration, Package Loader, Data Model, Archive content
ardumont renamed T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 from production: Deploy package loader v1.0, deposit server v0.16, lister v2.3 to production: Deploy package loader v1.1, deposit server v0.16, lister v2.3.
Mon, Nov 22, 1:06 PM · System administration, Package Loader, Data Model, Archive content
vlorentz placed T1260: Extend the release object model to allow synthetic objects up for grabs.
Mon, Nov 22, 12:05 PM · Archive content
ardumont triaged T3745: production: Deploy package loader v1.1, deposit server v0.16, lister v2.3 as Normal priority.
Mon, Nov 22, 11:30 AM · System administration, Package Loader, Data Model, Archive content

Wed, Nov 10

ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, as Resolved.
Wed, Nov 10, 4:43 PM · Package Loader, Data Model, Archive content
ardumont closed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Resolved.
Wed, Nov 10, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont placed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 up for grabs.
Wed, Nov 10, 4:43 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from in-progress to deployed/landed on the System administration board.
Wed, Nov 10, 4:29 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont added a comment to T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.

At least loader deposit and npm [1] are fine.

Wed, Nov 10, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Wed, Nov 10, 4:24 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont moved T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Backlog to in-progress on the System administration board.
Wed, Nov 10, 3:35 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont renamed T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from staging: Deploy package loader v1.0 and deposit server v0.16 to staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Wed, Nov 10, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3, a subtask of T3638: Make package loaders create releases objects instead of revisions, from Open to Work in Progress.
Wed, Nov 10, 3:33 PM · Package Loader, Data Model, Archive content
ardumont changed the status of T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 from Open to Work in Progress.
Wed, Nov 10, 3:33 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Wed, Nov 10, 3:32 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz updated the task description for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3.
Wed, Nov 10, 3:21 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
vlorentz added a parent task for T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3: T3636: Make the opam loader write extrinsic metadata.
Wed, Nov 10, 3:20 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content
ardumont triaged T3722: staging: Deploy package loader v1.0, deposit server v0.16, lister v2.3 as Normal priority.
Wed, Nov 10, 3:17 PM · System administration, SWORD deposit, Package Loader, Data Model, Archive content

Tue, Nov 9

vlorentz added a revision to T3638: Make package loaders create releases objects instead of revisions: D6618: Document how each package loader populates fields..
Tue, Nov 9, 12:39 PM · Package Loader, Data Model, Archive content

Mon, Nov 8

vlorentz added revisions to T3638: Make package loaders create releases objects instead of revisions: D6616: Make package loaders write releases instead of revisions, D6617: Use release instead of revision as anchor in SWHID context instead..
Mon, Nov 8, 11:58 AM · Package Loader, Data Model, Archive content
vlorentz added a comment to T3638: Make package loaders create releases objects instead of revisions.

Here is an overview of the fields (+ internal version name + branch name) used by each package loader:

Mon, Nov 8, 11:50 AM · Package Loader, Data Model, Archive content

Oct 22 2021

vlorentz added revisions to T1258: Synthesize release objects for all upstream things that match the concept of a release: D6529: deposit: Remove 'parent' deposit, D6530: Remove unused 'known_artifacts' code.
Oct 22 2021, 3:45 PM · Archive content
vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

Great news: of the 469k corrupt SVN revisions, all but 14 (yes, 14) can be fixed simply by adding 1 microsecond to their timestamp.

Oct 22 2021, 2:33 PM · Archive content, Restricted Project

Oct 20 2021

vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

After further investigation, I can't find any directory that is in a completely bad order; they are either ordered like git does (by adding a / at the end of dir entries) or by assuming a null byte at the end of dir entries.

Oct 20 2021, 12:18 PM · Archive content, Restricted Project

Oct 15 2021

vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

analysis on directories (some are also part of the fixable_trivial above, but I don't have the exact number, I lost it in my analysis):

Oct 15 2021, 11:21 AM · Archive content, Restricted Project
zack updated subscribers of T3656: Survey revisions/releases with partially loaded history.
Oct 15 2021, 9:34 AM · Archive content
zack added a comment to T3656: Survey revisions/releases with partially loaded history.
In T3656#72364, @grouss wrote:

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)

Oct 15 2021, 9:33 AM · Archive content
grouss added a comment to T3656: Survey revisions/releases with partially loaded history.

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)
checking in the current live swh DAG 2 days ago 98% have one in release or snapshot_branch.
indeed I was surprised because I did'nt have to loop over the revision history.

Oct 15 2021, 9:25 AM · Archive content
olasd added a project to T3660: Nodes with missing ancestors in SWH DAG / SWH-graph: Archive content.
Oct 15 2021, 9:17 AM · Archive content
ardumont added a comment to T3656: Survey revisions/releases with partially loaded history.

You might be interested by what @grouss just opened in T3660
(ah scratched that, zack already mentioned it)

Oct 15 2021, 9:07 AM · Archive content
zack added a subtask for T3656: Survey revisions/releases with partially loaded history: T3660: Nodes with missing ancestors in SWH DAG / SWH-graph.
Oct 15 2021, 8:56 AM · Archive content
zack updated subscribers of T3656: Survey revisions/releases with partially loaded history.

In T3660, @grouss has found many more.
Might be for a different reason (the dataset he analyzed is not the live one), but it's worth a comparison.

Oct 15 2021, 8:55 AM · Archive content
olasd added a comment to T3656: Survey revisions/releases with partially loaded history.
21:57 guest@softwareheritage => select count(distinct id) from revision_history where not exists (select 1 from revision where id=parent_id);
 count 
───────
  2218
(1 ligne)
Oct 15 2021, 8:50 AM · Archive content

Oct 14 2021

olasd triaged T3656: Survey revisions/releases with partially loaded history as Low priority.
Oct 14 2021, 11:40 AM · Archive content

Oct 13 2021

vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

My script finished running on releases. Result: all 644k releases are recoverable (mostly just missing gpg signatures), except 75k whose origin does not exist anymore.

Oct 13 2021, 6:40 PM · Archive content, Restricted Project