Page MenuHomeSoftware Heritage
Create Task
Maniphest T3412

Deploy vault "git bare" tarball cooker (swh-vault v0.6)
Closed, MigratedEdits Locked
Actions

Description

This requires:

  • the swh.vault v0.6
  • some changes in the default configuration [1] so the vault benefits from using objstorage and graph [2] services.

First staging then production if everything is ok.

[1] P1077

[2] The graph service is not ready for prime time, it needs some packaging and internal reworking regarding the existing swh.graph.client module (possibly to be moved to swh.web.client)

Revisions and Commits

rDVAU Software Heritage Vault
D5940rDVAU066a98b61f53 git_bare: Add support for filtered content with Git >= 2.21
D5938rDVAU04b174efc79f Make swh.graph dependency optional 2/2
D5935rDVAU9d0df34d412a Make swh.graph dependency optional
rSPSITE puppet-swh-site
D5933rSPSITE57c17bb577e5 Add information about swh.graph service
D5933rSPSITE9d0c85a87362 Adapt production vault server configuration to use the objstorage

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT887 Vault: "snapshot" cooker
 Migratedgitlab-migrationT3096 Efficient and reliable download via the Vault
 Migratedgitlab-migrationT3504 Make the git-bare cooker publicly available
 Migratedgitlab-migrationT3505 Make the git-bare cooker available to the staff and beta-testers in the production webapp
 Migratedgitlab-migrationT843 Vault: Add a "git bare" tarball cooker
 Migratedgitlab-migrationT3412 Deploy vault "git bare" tarball cooker (swh-vault v0.6)
 Migratedgitlab-migrationT3551 Fix git-fsck errors in the git-bare cooker
 Migratedgitlab-migrationT3552 Fix corrupted releases, revisions, and directories in the storage
 Migratedgitlab-migrationT75 Check integrity of directories, revisions, and releases
 Migratedgitlab-migrationT366 Refactor the person table
 Migratedgitlab-migrationT3594 Faithfully store weird git objects
 Migratedgitlab-migrationT3595 Support disordered directory entries in git
 Migratedgitlab-migrationT3596 Support "weird" permissions in directories
 Migratedgitlab-migrationT3598 Support revisions with "extra headers" not at the end
 Migratedgitlab-migrationT3752 Store/represent time offsets as strings
 Migratedgitlab-migrationT3819 Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production
 Migratedgitlab-migrationT3753 Store original git manifests
 Migratedgitlab-migrationT3553 git_bare: Don't include submodule targets
 Migratedgitlab-migrationT3566 git-fsck errors in the git-bare cooker are not properly reported
 Migratedgitlab-migrationT3731 Try parallelizing directory_get_entries with an async swh-storage client
 Migratedgitlab-migrationT3924 Write mailmaps after cooking git-bare archives with display names?

Event Timeline

ardumont triaged this task as Normal priority.Jun 28 2021, 12:26 PM
ardumont created this task.
ardumont mentioned this in Unknown Object (Paste).Jun 28 2021, 1:59 PM
ardumont updated the task description. (Show Details)
ardumont added a revision: D5933: Adapt production vault server configuration to use graph and objstorage.Jun 28 2021, 2:09 PM
ardumont updated the task description. (Show Details)Jun 28 2021, 2:13 PM
ardumont mentioned this in rSPSITEc77e9c13e7cc: worker3.staging: Add vault cooker service.Jun 28 2021, 2:37 PM
ardumont added a comment.Jun 28 2021, 2:56 PM

Debian build is broken [1], currently investigating to reproduce/fix it.

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DVAU/job/gbp-buildpackage/53/console

ardumont updated the task description. (Show Details)Jun 28 2021, 3:13 PM
ardumont removed a project: Vault.
ardumont added a project: Vault.
ardumont updated the task description. (Show Details)Jun 28 2021, 3:26 PM
ardumont added a revision: D5935: Make swh.graph dependency optional.Jun 28 2021, 3:38 PM
ardumont added a commit: rDVAU9d0df34d412a: Make swh.graph dependency optional.Jun 29 2021, 10:11 AM
ardumont added a comment.EditedJun 29 2021, 10:16 AM

Debian build is broken [1], currently investigating to reproduce/fix it.

Well, swh.graph is not debian packaged yet so the build was missing that dependency.

The path forward was either:

  • 1. at least build one debian package for swh.graph.client module (that's the part the vault requires at the moment)
  • 2. make the dependency on swh.graph optional (because the vault is opt-in for using it)

So, as there has been discussion about moving the swh.graph.client python code part to
swh.web.client [1], the solution was to not depend on it yet. Thus the solution 2. got
selected [2].

[1] P1079

[2] D5935 D5938

ardumont added a revision: D5938: Make swh.graph dependency optional 2/2.Jun 29 2021, 10:35 AM
ardumont added a commit: rDVAU04b174efc79f: Make swh.graph dependency optional 2/2.Jun 29 2021, 11:08 AM
ardumont added a comment.Jun 29 2021, 11:24 AM

The build is still breaking in a different way [1]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DVAU/job/gbp-buildpackage/55/console

ardumont added a revision: D5940: git_bare: Add support for filtered content with Git >= 2.21.Jun 29 2021, 11:41 AM
vlorentz added a commit: rDVAU066a98b61f53: git_bare: Add support for filtered content with Git >= 2.21.Jun 29 2021, 11:54 AM
ardumont added a comment.EditedJun 29 2021, 12:08 PM

status:

  • unstable build is fine.
  • stable build failed [1] [2]

This task feels like more and more like a boring whack a mole game.

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DVAU/job/gbp-buildpackage/57/console

[2] P1082

ardumont added a commit: rSPSITE9d0c85a87362: Adapt production vault server configuration to use the objstorage.Jun 29 2021, 1:32 PM
ardumont added a commit: rSPSITE57c17bb577e5: Add information about swh.graph service.
ardumont added a project: System administration.Jun 29 2021, 1:44 PM
ardumont moved this task from Backlog to deployed/landed/monitoring on the System administration board.
ardumont updated the task description. (Show Details)
ardumont added a comment.EditedJun 29 2021, 1:48 PM

Deployed v0.6.4.

ardumont closed this task as Resolved.Jun 29 2021, 1:54 PM
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
vlorentz renamed this task from Deploy new vault "git bare" tarball cooker to Deploy vault "git bare" tarball cooker (swh-vault v0.6).Jun 29 2021, 2:02 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:03 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: vlorentz, gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal