Page MenuHomeSoftware Heritage
Create Task
Maniphest T565

embrace repository snapshot object in the data model (meta task)
Closed, MigratedEdits Locked
Actions

Description

(Multiple) origins might expose the very same state of a repository over time.
To both avoid bloat and be more coherent with the "big Merkle DAG" approach of the data model, we want to introduce a new kind of node in our DAGs that capture the current state of a repository and associate to it a content-adressable identifier.

These new Merkle DAG nodes will be called repository snapshot objects.
A more formal description of repository snapshots is available in

repository-snapshot-object.md4 KBDownload
and on the wiki.

This meta task tracks the various changes needed throughout Software Heritage to fully embrace repository snapshot objects.

Related Objects
Search...

Event Timeline

zack triaged this task as Normal priority.Sep 15 2016, 4:26 PM
zack created this task.
zack updated the task description. (Show Details)
zack created subtask T566: specify serialization format for repository snapshot objects.Sep 15 2016, 4:33 PM
zack created subtask T567: adapt SQL storage for repository snapshot objects.
seirl added a parent task: T531: Vault cookers.Aug 4 2017, 4:27 PM
olasd added a subtask: T830: Remove tables occurrence and occurrence_history.Nov 5 2017, 9:29 PM
ardumont mentioned this in D268: Add snapshot models.Nov 14 2017, 11:28 AM
seirl added a parent task: T887: Vault: "snapshot" cooker.Dec 12 2017, 3:45 PM
olasd closed subtask T566: specify serialization format for repository snapshot objects as Resolved.Dec 13 2017, 11:30 AM
olasd closed subtask T567: adapt SQL storage for repository snapshot objects as Resolved.Dec 15 2017, 3:37 PM
ardumont mentioned this in rDLDSVN666f838a34c4: loader: replace occurrences with snapshot.Jan 31 2018, 3:40 PM
ardumont mentioned this in rDLDSVNe549e173035d: loader: Start back from latest visit through snapshot.Jan 31 2018, 5:32 PM
ardumont mentioned this in rDLDDIRa5e626279cc4: loader: replace occurrences with snapshot.Jan 31 2018, 6:28 PM
ardumont mentioned this in rDLDTAR7b24c1ab3bef: loader: replace occurrences with snapshot.Jan 31 2018, 6:36 PM
ardumont mentioned this in rDLDSVN27edc30e4bf5: loader: Make uneventful visit reference the latest snapshot.Feb 1 2018, 3:12 PM
ardumont mentioned this in rDDEP7e953f6f49f7: swh.deposit.loader: Replace occurrences with snapshot.Feb 2 2018, 9:13 AM
ardumont mentioned this in rDLDHG394f178b5ab6: swh.mercurial.loader: Replace occurrences with snapshot.Feb 2 2018, 10:04 AM
ardumont mentioned this in rDLDHGc5b0059a29a5: swh.mercurial.loader: Fix slow_loader + replace occs with snapshot.Feb 2 2018, 11:01 AM
ardumont added a subscriber: ardumont.EditedFeb 2 2018, 11:19 AM

Current status on the development migration towards snapshot (branch wip/snapshot(s)) as far as I know:

  • swh-storage (opened endpoints to store/retrieve)
  • swh-model (expose identifiers functions to compute hash)
  • swh-loader-dir
  • swh-loader-tar
  • swh-loader-deposit
  • swh-loader-svn
  • swh-loader-mercurial
  • swh-loader-git
  • swh-loader-debian
  • swh-web
ardumont mentioned this in D297: swh.web: Add snapshot conversion on origin_visit.Feb 6 2018, 12:52 PM
ardumont created subtask T949: swh-web: Display origin-visit's details using snapshots.Feb 6 2018, 1:51 PM
ardumont mentioned this in rDWAPPS8e8df4bbb6db: swh.web.api: Add snapshot conversion on origin_visit.Feb 6 2018, 1:56 PM
ardumont mentioned this in rSPSITEfea325a23832: data/defaults: Update and remove unneeded properties.Feb 6 2018, 3:24 PM
olasd added a subscriber: olasd.Feb 6 2018, 4:30 PM

swh-loader-git and swh-loader-debian have now been migrated to snapshots as well, and restarted.

fiendish mentioned this in rDLDHGc2082f52fefe: swh.mercurial.loader: Replace occurrences with snapshot.Feb 8 2018, 8:12 PM
fiendish mentioned this in rDLDHG4e4f019d0cde: swh.mercurial.loader: Fix slow_loader + replace occs with snapshot.
ardumont mentioned this in rDLDTAR7738bdb205af: loader: replace occurrences with snapshot.Feb 12 2018, 11:20 AM
ardumont mentioned this in rDDEPcd80d1cce241: d/control: Bump requirements up for new swh.loader.tar.Feb 12 2018, 11:22 AM
ardumont mentioned this in rDLDHGf098831fd810: bundle20_loader: snapshot: Use swh identifier in branch target.Feb 14 2018, 6:26 PM
ardumont mentioned this in rDLDHGda9f4bade658: bundle20_loader: snapshot: Target releases if any.
ardumont mentioned this in rDLDHGefc9a13abf75: bundle20_loader: Acknowledge we are missing the default branch.
ardumont mentioned this in rDLDHGb9a888fec10a: bundle20_loader: Read heads to create snapshot.
ardumont mentioned this in rDLDHGc2f23f452608: bundle20_loader: Use swh revision id when creating the snapshot.Feb 15 2018, 10:05 AM
ardumont mentioned this in rDLDHG6e8a1862fb25: bundle20_loader: Read bookmark as branch and use it within snapshot.Feb 15 2018, 2:49 PM
anlambert closed subtask T949: swh-web: Display origin-visit's details using snapshots as Resolved.Feb 15 2018, 3:35 PM
olasd closed subtask T830: Remove tables occurrence and occurrence_history as Resolved.Oct 16 2018, 12:20 PM
ardumont mentioned this in R65:8e8df4bbb6db: swh.web.api: Add snapshot conversion on origin_visit.Feb 23 2019, 1:56 AM
zack closed this task as Resolved.Mar 12 2019, 10:10 AM
zack claimed this task.

unless i'm missing something, this has been completed a while ago (if not, please reopen, ideally adding the relevant open sub-task)

gitlab-migration changed the status of subtask T566: specify serialization format for repository snapshot objects from Resolved to Migrated.Jan 8 2023, 4:19 PM
gitlab-migration changed the status of subtask T567: adapt SQL storage for repository snapshot objects from Resolved to Migrated.
gitlab-migration changed the status of subtask T949: swh-web: Display origin-visit's details using snapshots from Resolved to Migrated.Jan 8 2023, 4:24 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 9:56 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T830: Remove tables occurrence and occurrence_history from Resolved to Migrated.Jan 8 2023, 9:57 PM
About · Developer information · Legal