Page MenuHomeSoftware Heritage

WIP git bare revision cooker
Needs ReviewPublic

Authored by tenma on Dec 18 2020, 10:59 AM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Summary

First version of the bare cooker, RevisionBareCooker, which only handles one revision history (one revision and its parents) and outputs a tarball.

This naive cooker still needs more work to get better performance. The main element being: it must check before fetching dirs and contents if they will be duplicate.

Related to T843

Test Plan

First, manual tests in docker environment.

Then a pytest test suite inspired by the other cookers.

Diff Detail

Event Timeline

Build has FAILED

Patch application report for D4766 (id=16864)

Rebasing onto 1b6a10fded...

Current branch diff-target is up to date.
Changes applied before test
commit 54019acfb25ccc1f9260419c79814e05f1049058
Author: tenma <tenma+swh@mailbox.org>
Date:   Fri Dec 18 10:57:27 2020 +0100

    WIP git bare revision cooker

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/61/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/61/console

Build has FAILED

Patch application report for D4766 (id=16865)

Rebasing onto 1b6a10fded...

Current branch diff-target is up to date.
Changes applied before test
commit 1ef446b3d5269b9b4a7ad2c9e873dd5a7f11d1d5
Author: tenma <tenma+swh@mailbox.org>
Date:   Fri Dec 18 10:57:27 2020 +0100

    WIP git bare revision cooker

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/62/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/62/console

tenma edited the test plan for this revision. (Show Details)
douardda added inline comments.
swh/vault/cookers/revision_bare.py
1

note the copyright dates are wrong

29

why these imports in there (rather than at the top of the module)?

tenma edited the test plan for this revision. (Show Details)
  • use directory_ls rather than dir_iterator to lower overhead and control iteration
  • use a set to test early for known objects
  • only update once master head with the given revision id
  • do not chunk files ids because in the end the storage requests are not chunked
  • partition new and old files to adapt processing

Build has FAILED

Patch application report for D4766 (id=17066)

Rebasing onto b2e17a95fe...

Current branch diff-target is up to date.
Changes applied before test
commit 0210ed0b79c531b4769ad8803160441777b708b6
Author: tenma <tenma+swh@mailbox.org>
Date:   Fri Dec 18 10:57:27 2020 +0100

    WIP git bare revision cooker

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/63/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/63/console

tenma marked 2 inline comments as done.Wed, Jan 6, 8:50 PM