Differential D5703

Run all directory tests on the gitfast cooker
ClosedPublic
Actions

Authored by vlorentz on May 6 2021, 2:43 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Commits

rDVAU545246e9af45: git_bare: When possible, use swh-graph instead of swh-storage to query revision…

Summary

It increases test coverage
test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

Diff Detail

Repository

rDVAU Software Heritage Vault

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.May 6 2021, 2:43 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMay 6 2021, 2:43 PM

Build has FAILED

Patch application report for D5703 (id=20352)

Could not rebase; Attempt merge onto 3c14013bc7...

Updating 3c14013..8007936
Fast-forward
 requirements-swh.txt            |   2 +-
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 +++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/conftest.py     |   4 +-
 swh/vault/tests/test_cookers.py | 217 +++++++++++++++++++++++++++-------------
 8 files changed, 373 insertions(+), 76 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py

Changes applied before test

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

commit 35c9f519cd705f67006037dcf8cb85c7129b0c7e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:25:27 2021 +0200

    Use swh.core 0.14
    
    It renamed db_name to dbname, which is a breaking change.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/80/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/80/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 6 2021, 2:44 PM

Harbormaster failed remote builds in B21334: Diff 20352!

vlorentz requested review of this revision.May 6 2021, 2:46 PM

vlorentz added a parent revision: D5658: git_bare: Deduplicate object downloads and writes.

olasd accepted this revision.May 11 2021, 12:39 PM

This revision is now accepted and ready to land.May 11 2021, 12:39 PM

Build has FAILED

Patch application report for D5703 (id=20352)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..8007936
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 +++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 217 +++++++++++++++++++++++++++-------------
 6 files changed, 370 insertions(+), 73 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py

Changes applied before test

commit 8007936a8aff0b29eefdd93bbe037b996d6b743d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 3e76bc5656d0aa1eb510dcfdaa3b6196f6ee5976
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/104/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/104/console

rebase

Build has FAILED

Patch application report for D5703 (id=20504)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..3db419a
Fast-forward
 swh/vault/cli.py                |   2 +-
 swh/vault/cookers/__init__.py   |   3 +
 swh/vault/cookers/base.py       |   6 +-
 swh/vault/cookers/git_bare.py   | 213 +++++++++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py  |   2 +-
 swh/vault/tests/test_cookers.py | 217 +++++++++++++++++++++++++++-------------
 6 files changed, 370 insertions(+), 73 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py

Changes applied before test

commit 3db419ac0a453277107fd51a2100b891675064c9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 19727ac48bd41dccdb0b73d08e444a232b85a5ad
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/107/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/107/console

Harbormaster failed remote builds in B21478: Diff 20504!May 11 2021, 3:29 PM

rebase

Build is green

Patch application report for D5703 (id=20508)

Could not rebase; Attempt merge onto 35c9f519cd...

Updating 35c9f51..545246e
Fast-forward
 requirements-swh.txt                    |   1 +
 swh/vault/cli.py                        |   6 +-
 swh/vault/cookers/__init__.py           |   6 +
 swh/vault/cookers/base.py               |  15 +-
 swh/vault/cookers/git_bare.py           | 252 ++++++++++++++++++++++++++++++++
 swh/vault/in_memory_backend.py          |   2 +-
 swh/vault/tests/test_cli.py             |   1 +
 swh/vault/tests/test_cookers.py         | 217 ++++++++++++++++++---------
 swh/vault/tests/test_git_bare_cooker.py | 181 +++++++++++++++++++++++
 9 files changed, 606 insertions(+), 75 deletions(-)
 create mode 100644 swh/vault/cookers/git_bare.py
 create mode 100644 swh/vault/tests/test_git_bare_cooker.py

Changes applied before test

commit 545246e9af45d98e6c4f741414460aef58201a7b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri May 7 11:30:10 2021 +0200

    git_bare: When possible, use swh-graph instead of swh-storage to query revision history
    
    We expect it to be more efficient eventually; but run time is equivalent so far.

commit 7494b111937d4a6e3f8cbbf37a674d4ef9a04de4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu May 6 14:43:37 2021 +0200

    Run all directory tests on the gitfast cooker
    
    1. It increases test coverage
    2. test_revision_bogus_perms it now redundant (there is test_directory_bogus_perms)

commit 891092d56ea7a8f89d5046dafc6da0d7fe8a2a97
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 22:22:17 2021 +0200

    git_bare: Deduplicate object downloads and writes

commit 4052f53698454ac47a01d26d470b8ab4b0f77a6d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/111/ for more details.

Harbormaster completed remote builds in B21482: Diff 20508.May 11 2021, 3:51 PM

Closed by commit rDVAU545246e9af45: git_bare: When possible, use swh-graph instead of swh-storage to query revision… (authored by vlorentz). · Explain WhyMay 11 2021, 3:54 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDVAU545246e9af45: git_bare: When possible, use swh-graph instead of swh-storage to query revision….