Page MenuHomeSoftware Heritage
Differential D5708

git_bare: When possible, use swh-graph instead of swh-storage to query revision history
ClosedPublic
Actions

Authored by vlorentz on May 7 2021, 11:30 AM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Summary

We expect it to be more efficient eventually; but run time is equivalent so far.

Depends on D5658 and D5665.

Test Plan

The test is painful to read or write. Suggestions welcome.

CI fails because it depends on unreleased swh-model and swh-graph features

Diff Detail

Event Timeline

vlorentz created this revision.May 7 2021, 11:30 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 7 2021, 11:30 AM
vlorentz edited the test plan for this revision. (Show Details)May 7 2021, 11:31 AM
swh-public-ci added a comment.May 7 2021, 11:31 AM

Build has FAILED

Patch application report for D5708 (id=20378)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..fee0f86
Fast-forward
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |   9 +-
 swh/vault/cookers/git_bare.py           | 246 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cookers.py         | 217 +++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 159 +++++++++++++++++++++
 7 files changed, 570 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit fee0f86b3cd7b0df9ee4eeb01b33ec6d106d09bb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/84/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/84/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 7 2021, 11:31 AM
Harbormaster failed remote builds in B21357: Diff 20378!
vlorentz updated this revision to Diff 20379.May 7 2021, 11:37 AM

fix case of revision not found in swh-graph

swh-public-ci added a comment.May 7 2021, 11:38 AM

Build has FAILED

Patch application report for D5708 (id=20379)

Rebasing onto 35c9f519cd...

Current branch diff-target is up to date.
Changes applied before test
commit 0063cae0d6bf9a882ccb6bef71342aaa1ca04ba7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/85/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/85/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 7 2021, 11:38 AM
Harbormaster failed remote builds in B21358: Diff 20379!
vlorentz edited the test plan for this revision. (Show Details)May 7 2021, 11:39 AM
vlorentz updated this revision to Diff 20380.May 7 2021, 11:39 AM

add dep on swh-graph

swh-public-ci added a comment.May 7 2021, 11:40 AM

Build has FAILED

Patch application report for D5708 (id=20380)

Rebasing onto 35c9f519cd...

Current branch diff-target is up to date.
Changes applied before test
commit e7fa2776cd1f0b3eab56cb8968293316c92e9c9e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/86/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/86/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 7 2021, 11:40 AM
Harbormaster failed remote builds in B21359: Diff 20380!
vlorentz updated this revision to Diff 20387.May 7 2021, 12:39 PM

fix tests

swh-public-ci added a comment.May 7 2021, 12:39 PM

Build has FAILED

Patch application report for D5708 (id=20387)

Rebasing onto 35c9f519cd...

Current branch diff-target is up to date.
Changes applied before test
commit 8805ca46bca8fd78aebc9684aa765136dd16ed87
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/87/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/87/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 7 2021, 12:39 PM
Harbormaster failed remote builds in B21366: Diff 20387!
vlorentz requested review of this revision.May 7 2021, 12:40 PM
vlorentz updated this revision to Diff 20388.May 7 2021, 12:40 PM

wrong base commit

swh-public-ci added a comment.May 7 2021, 12:41 PM

Build has FAILED

Patch application report for D5708 (id=20388)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..8805ca4
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 178 ++++++++++++++++++++++
 9 files changed, 603 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit 8805ca46bca8fd78aebc9684aa765136dd16ed87
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/88/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/88/console

Harbormaster failed remote builds in B21367: Diff 20388!May 7 2021, 12:41 PM
vlorentz added a task: T843: Vault: Add a "git bare" tarball cooker.May 10 2021, 9:48 AM
vlorentz updated this revision to Diff 20460.May 10 2021, 7:34 PM

proper check of swhid existence

swh-public-ci added a comment.May 10 2021, 7:35 PM

Build has FAILED

Patch application report for D5708 (id=20460)

Rebasing onto 35c9f519cd...

Current branch diff-target is up to date.
Changes applied before test
commit 43e735a7a5fc8c7f89275df8a03124358c0c3cc3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/89/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/89/console

Harbormaster failed remote builds in B21435: Diff 20460!May 10 2021, 7:35 PM
vlorentz added a child revision: D5723: Add endpoint directory_get_entries, to quickly list a directory's entries.May 10 2021, 9:53 PM
vlorentz added a child revision: D5730: git_bare: Use batched content_get() instead of content_find().
vlorentz added a child revision: D5733: git_bare: Refactor the graph descent using explicit stacks instead of the call stack..May 11 2021, 9:57 AM
olasd mentioned this in D5733: git_bare: Refactor the graph descent using explicit stacks instead of the call stack..May 11 2021, 11:29 AM
vlorentz updated this revision to Diff 20484.May 11 2021, 11:41 AM

remove unwanted commits

swh-public-ci added a comment.May 11 2021, 11:41 AM

Build has FAILED

Patch application report for D5708 (id=20484)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..43e735a
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 178 ++++++++++++++++++++++
 9 files changed, 603 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit 43e735a7a5fc8c7f89275df8a03124358c0c3cc3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/98/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/98/console

Harbormaster failed remote builds in B21457: Diff 20484!May 11 2021, 11:41 AM
olasd accepted this revision.May 11 2021, 12:14 PM
olasd added a subscriber: olasd.

Ah, now I understand why the swh.graph query is restricted to rev:rev edges. I guess it makes sense within that version.

swh/vault/cookers/git_bare.py
86

Sneaky hunk from another commit, I guess (no need for re-review)

swh/vault/tests/test_git_bare_cooker.py
107–138 ↗(On Diff #20484)

As this initialization really depends on the shape of the objects, it would probably be clearer if it was wrapped in the same function as the one which creates the objects in the first place. You could put the population of the edge lists next to the object initialization.

This revision is now accepted and ready to land.May 11 2021, 12:14 PM
vlorentz updated this revision to Diff 20496.May 11 2021, 12:29 PM

refactor test as requested

swh-public-ci added a comment.May 11 2021, 12:30 PM

Build has FAILED

Patch application report for D5708 (id=20496)

Rebasing onto 35c9f519cd...

Current branch diff-target is up to date.
Changes applied before test
commit 2ec60e27c75775f7073dd51947648a999748be35
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/99/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/99/console

Harbormaster failed remote builds in B21470: Diff 20496!May 11 2021, 12:30 PM
vlorentz updated this revision to Diff 20497.May 11 2021, 12:30 PM

wrong commits

swh-public-ci added a comment.May 11 2021, 12:31 PM

Build has FAILED

Patch application report for D5708 (id=20497)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..2ec60e2
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 181 +++++++++++++++++++++++
 9 files changed, 606 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit 2ec60e27c75775f7073dd51947648a999748be35
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/100/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/100/console

Harbormaster failed remote builds in B21471: Diff 20497!May 11 2021, 12:31 PM
swh-public-ci added a comment.May 11 2021, 3:26 PM

Build has FAILED

Patch application report for D5708 (id=20497)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..2ec60e2
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 181 +++++++++++++++++++++++
 9 files changed, 606 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit 2ec60e27c75775f7073dd51947648a999748be35
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/105/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/105/console

vlorentz updated this revision to Diff 20505.May 11 2021, 3:26 PM

rebase

swh-public-ci added a comment.May 11 2021, 3:29 PM

Build has FAILED

Patch application report for D5708 (id=20505)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..50b9ed6
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 181 +++++++++++++++++++++++
 9 files changed, 606 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit 50b9ed6586d2bc649c93e054c1392d2e7fb8d891
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 3db419ac0a453277107fd51a2100b891675064c9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 19727ac48bd41dccdb0b73d08e444a232b85a5ad
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/108/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/108/console

Harbormaster failed remote builds in B21479: Diff 20505!May 11 2021, 3:29 PM
vlorentz updated this revision to Diff 20509.May 11 2021, 3:46 PM

rebase

swh-public-ci added a comment.May 11 2021, 3:53 PM

Build is green

Patch application report for D5708 (id=20509)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..e77069a
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 279 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 +++++++++++++++++--------
 swh/vault/tests/test_git_bare_cooker.py | 181 +++++++++++++++++++++
 9 files changed, 633 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py
Changes applied before test
commit e77069a5e1cab7630e912ac59b0aa1242346a95f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 20:09:30 2021 +0200

    git_bare: Refactor the graph descent using explicit stacks instead of the call stack.
    
    This will allow batching large groups of objects, instead of being limited
    to those given as argument from a parent.

commit 545246e9af45d98e6c4f741414460aef58201a7b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 7494b111937d4a6e3f8cbbf37a674d4ef9a04de4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 891092d56ea7a8f89d5046dafc6da0d7fe8a2a97
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/112/ for more details.

Harbormaster completed remote builds in B21483: Diff 20509.May 11 2021, 3:53 PM
vlorentz closed this revision.May 11 2021, 3:57 PM

545246e9af45d98e6c4f741414460aef58201a7b

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
vault/
cookers/
115 lines97%
Loading...

Diff 20509

View Options

swh/vault/cookers/git_bare.py

Loading...
About · Developer information · Legal