Page MenuHomeSoftware Heritage
Differential D5658

git_bare: Deduplicate object downloads and writes
ClosedPublic
Actions

Authored by vlorentz on Apr 30 2021, 10:24 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Commits
rDVAU7494b111937d: Run all directory tests on the gitfast cooker

Diff Detail

Event Timeline

vlorentz created this revision.Apr 30 2021, 10:24 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 30 2021, 10:24 PM
swh-public-ci added a comment.Apr 30 2021, 10:25 PM

Build has FAILED

Patch application report for D5658 (id=20211)

Could not rebase; Attempt merge onto f87dd54d5d...

Updating f87dd54..418db8e
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 ++++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 135 ++++++++++++++++++++-----
 6 files changed, 334 insertions(+), 27 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit 418db8e87279092125c287a8f2cb528abd5d4619
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit c3984d34c2a8cf32ff63595013b64860f1e25054
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/73/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/73/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 30 2021, 10:25 PM
Harbormaster failed remote builds in B21201: Diff 20211!
vlorentz added a parent revision: D5656: Add a naive git bare cooker.Apr 30 2021, 11:10 PM
vlorentz requested review of this revision.May 3 2021, 10:59 AM
vlorentz updated this revision to Diff 20271.May 4 2021, 6:49 PM

rebase

swh-public-ci added a comment.May 4 2021, 6:50 PM

Build has FAILED

Patch application report for D5658 (id=20271)

Could not rebase; Attempt merge onto f87dd54d5d...

Updating f87dd54..c7f77b8
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 ++++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 135 ++++++++++++++++++++-----
 6 files changed, 334 insertions(+), 27 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit c7f77b8921173b4b4f5eb18687780ace7d387237
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 91472855a218e6ebcd81b496936f6f8b234fde38
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/75/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/75/console

Harbormaster failed remote builds in B21258: Diff 20271!May 4 2021, 6:50 PM
vlorentz added a child revision: D5703: Run all directory tests on the gitfast cooker.May 6 2021, 2:46 PM
vlorentz added a child revision: D5708: git_bare: When possible, use swh-graph instead of swh-storage to query revision history.May 7 2021, 11:30 AM
vlorentz added a task: T843: Vault: Add a "git bare" tarball cooker.May 10 2021, 9:48 AM
douardda added a subscriber: douardda.May 11 2021, 9:48 AM
douardda added inline comments.
swh/vault/cookers/git_bare.py
175

this 10240 comes from somewhere? or it's just a reasonable random value?

vlorentz added inline comments.May 11 2021, 9:54 AM
swh/vault/cookers/git_bare.py
175

the latter. It's only temporary anyway, I have a commit locally that gets rid of @functools.lru_cache.

vlorentz added inline comments.May 11 2021, 9:59 AM
swh/vault/cookers/git_bare.py
175

It's in D5733

olasd accepted this revision.May 11 2021, 11:33 AM
olasd added a subscriber: olasd.

(I'm not sure getting rid of the lru cache in favor of keeping the full set in memory is wise, for very large bundles. We'll see when we get there)

This revision is now accepted and ready to land.May 11 2021, 11:33 AM
swh-public-ci added a comment.May 11 2021, 3:23 PM

Build has FAILED

Patch application report for D5658 (id=20271)

Could not rebase; Attempt merge onto 35c9f519cd...

Auto-merging swh/vault/tests/test_cookers.py
Merge made by the 'recursive' strategy.
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 ++++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 135 ++++++++++++++++++++-----
 6 files changed, 334 insertions(+), 27 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit cda76fff3b5ac5accd9d2105df6926f4276d865a
Merge: 35c9f51 c7f77b8
Author: Jenkins user <jenkins@localhost>
Date:   Tue May 11 13:22:08 2021 +0000

    Merge branch 'diff-target' into HEAD

commit c7f77b8921173b4b4f5eb18687780ace7d387237
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 91472855a218e6ebcd81b496936f6f8b234fde38
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/103/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/103/console

vlorentz updated this revision to Diff 20503.May 11 2021, 3:26 PM

fix docstring

swh-public-ci added a comment.May 11 2021, 3:27 PM

Build has FAILED

Patch application report for D5658 (id=20503)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..19727ac
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 ++++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 135 ++++++++++++++++++++-----
 6 files changed, 334 insertions(+), 27 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit 19727ac48bd41dccdb0b73d08e444a232b85a5ad
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/106/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/106/console

Harbormaster failed remote builds in B21477: Diff 20503!May 11 2021, 3:27 PM
vlorentz updated this revision to Diff 20506.May 11 2021, 3:45 PM

fix again

vlorentz updated this revision to Diff 20507.May 11 2021, 3:45 PM

fix again

swh-public-ci added a comment.May 11 2021, 3:47 PM

Build is green

Patch application report for D5658 (id=20506)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..891092d
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 ++++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 135 ++++++++++++++++++++-----
 6 files changed, 334 insertions(+), 27 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit 891092d56ea7a8f89d5046dafc6da0d7fe8a2a97
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/109/ for more details.

Harbormaster completed remote builds in B21480: Diff 20506.May 11 2021, 3:47 PM
swh-public-ci added a comment.May 11 2021, 3:49 PM

Build is green

Patch application report for D5658 (id=20507)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..7494b11
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 +++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 217 +++++++++++++++++++++++++++-------------
 6 files changed, 370 insertions(+), 73 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
Changes applied before test
commit 7494b111937d4a6e3f8cbbf37a674d4ef9a04de4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 891092d56ea7a8f89d5046dafc6da0d7fe8a2a97
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/110/ for more details.

Harbormaster completed remote builds in B21481: Diff 20507.May 11 2021, 3:49 PM
Closed by commit rDVAU7494b111937d: Run all directory tests on the gitfast cooker (authored by vlorentz). · Explain WhyMay 11 2021, 3:54 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDVAU7494b111937d: Run all directory tests on the gitfast cooker.

Revision Contents
Changeset List

PathSize
swh/
vault/
cookers/
51 lines

Diff 20503

View Options

swh/vault/cookers/git_bare.py

Loading...
About · Developer information · Legal