Page MenuHomeSoftware Heritage

Add a naive git bare cooker
ClosedPublic

Authored by vlorentz on Fri, Apr 30, 7:32 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Summary

It can cook directories (by adding a synthetic revision pointing to it)
and revisions.

Current limitations:

  • It does not deduplicate directories and files at all, and queries all objects one by one.
  • No support for missing/absent contents
  • No support for missing submodules

Tests reuse existing tests of the DirectoryCooker and
RevisionGitfastCooker using parametrized pytest fixtures.

Event Timeline

Build has FAILED

Patch application report for D5656 (id=20205)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit caeddd4492e4888598e47e4dfa87991e4db77853
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/console

Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Apr 30, 7:32 PM
Harbormaster failed remote builds in B21195: Diff 20205!

Build has FAILED

Patch application report for D5656 (id=20206)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit c3984d34c2a8cf32ff63595013b64860f1e25054
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/console

Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Apr 30, 7:34 PM
Harbormaster failed remote builds in B21196: Diff 20206!
douardda added inline comments.
swh/vault/cookers/git_bare.py
87

As said on IRC, I don't think this 'directory' support should be in there. If it's only purpose is to help reusing tests, then tests should be refactored instead of adding code in the business logic.

111

Maybe we should not use an hardcoded 'mater' branch here? we should at least make it configurable (nowadays)?

134

why in "append" mode ?

179

do you mean objects like SkippedContent?

swh/vault/cookers/git_bare.py
87

It's also useful when testing manually with the CLI.

111

We can make it configurable when we'll need it.

179

"filtered" is the generic name for "skipped or hidden or missing"

don't use append mode for writing objects

Build has FAILED

Patch application report for D5656 (id=20270)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit 91472855a218e6ebcd81b496936f6f8b234fde38
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/console

douardda added inline comments.
swh/vault/cookers/git_bare.py
87

I don't really understand why (what is really the added value of this git-bare-directory' cooker), but do as you wish.

179

indeed

This revision is now accepted and ready to land.Wed, May 5, 12:40 PM

Build is green

Patch application report for D5656 (id=20270)

Rebasing onto 35c9f519cd...

First, rewinding head to replay your work on top of it...
Applying: Add a naive git bare cooker
Changes applied before test
commit 69d84aa2240a18e702892c2fe59f731d72c5d50e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/102/ for more details.