Page MenuHomeSoftware Heritage
Differential D5656

Add a naive git bare cooker
ClosedPublic
Actions

Authored by vlorentz on Apr 30 2021, 7:32 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Summary

It can cook directories (by adding a synthetic revision pointing to it)
and revisions.

Current limitations:

  • It does not deduplicate directories and files at all, and queries all objects one by one.
  • No support for missing/absent contents
  • No support for missing submodules

Tests reuse existing tests of the DirectoryCooker and
RevisionGitfastCooker using parametrized pytest fixtures.

Diff Detail

Repository
rDVAU Software Heritage Vault
Branch
git-bare-cooker
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 21196
Build 32903: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 32902: arc lint + arc unit

Event Timeline

vlorentz created this revision.Apr 30 2021, 7:32 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 30 2021, 7:32 PM
swh-public-ci added a comment.Apr 30 2021, 7:32 PM

Build has FAILED

Patch application report for D5656 (id=20205)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit caeddd4492e4888598e47e4dfa87991e4db77853
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 30 2021, 7:32 PM
Harbormaster failed remote builds in B21195: Diff 20205!
vlorentz updated this revision to Diff 20206.Apr 30 2021, 7:33 PM

oops, tempdir

swh-public-ci added a comment.Apr 30 2021, 7:34 PM

Build has FAILED

Patch application report for D5656 (id=20206)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit c3984d34c2a8cf32ff63595013b64860f1e25054
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 30 2021, 7:34 PM
Harbormaster failed remote builds in B21196: Diff 20206!
vlorentz added a child revision: D5658: git_bare: Deduplicate object downloads and writes.Apr 30 2021, 11:10 PM
vlorentz requested review of this revision.May 3 2021, 10:59 AM
douardda added a subscriber: douardda.May 4 2021, 6:17 PM
douardda added inline comments.
swh/vault/cookers/git_bare.py
87

As said on IRC, I don't think this 'directory' support should be in there. If it's only purpose is to help reusing tests, then tests should be refactored instead of adding code in the business logic.

111

Maybe we should not use an hardcoded 'mater' branch here? we should at least make it configurable (nowadays)?

134

why in "append" mode ?

179

do you mean objects like SkippedContent?

vlorentz added inline comments.May 4 2021, 6:44 PM
swh/vault/cookers/git_bare.py
87

It's also useful when testing manually with the CLI.

111

We can make it configurable when we'll need it.

179

"filtered" is the generic name for "skipped or hidden or missing"

vlorentz updated this revision to Diff 20270.May 4 2021, 6:48 PM

don't use append mode for writing objects

swh-public-ci added a comment.May 4 2021, 6:49 PM

Build has FAILED

Patch application report for D5656 (id=20270)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit 91472855a218e6ebcd81b496936f6f8b234fde38
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/console

Harbormaster failed remote builds in B21257: Diff 20270!May 4 2021, 6:49 PM
douardda accepted this revision.May 5 2021, 12:40 PM
douardda added inline comments.
swh/vault/cookers/git_bare.py
87

I don't really understand why (what is really the added value of this git-bare-directory' cooker), but do as you wish.

179

indeed

This revision is now accepted and ready to land.May 5 2021, 12:40 PM
vlorentz added a parent revision: D5652: identifiers: Expose git_object instead of manifest.May 6 2021, 2:46 PM
vlorentz added a task: T843: Vault: Add a "git bare" tarball cooker.May 10 2021, 9:48 AM
swh-public-ci added a comment.May 11 2021, 3:21 PM

Build is green

Patch application report for D5656 (id=20270)

Rebasing onto 35c9f519cd...

First, rewinding head to replay your work on top of it...
Applying: Add a naive git bare cooker
Changes applied before test
commit 69d84aa2240a18e702892c2fe59f731d72c5d50e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/102/ for more details.

Harbormaster completed remote builds in B21257: Diff 20270.May 11 2021, 3:22 PM
vlorentz closed this revision.May 11 2021, 3:57 PM

4052f53698454ac47a01d26d470b8ab4b0f77a6d

Revision Contents
Changeset List

PathSize
swh/
vault/
2 lines
cookers/
3 lines
6 lines
184 lines
2 lines
tests/
135 lines

Diff 20206

View Options

swh/vault/cli.py

Loading...
View Options

swh/vault/cookers/__init__.py

Loading...
View Options

swh/vault/cookers/base.py

Loading...
View Options

swh/vault/cookers/git_bare.py

Loading...
View Options

swh/vault/in_memory_backend.py

Loading...
View Options

swh/vault/tests/test_cookers.py

Loading...
About · Developer information · Legal