Page MenuHomeSoftware Heritage

Call objstorage.content_get() with a HashDict instead of single hash
Needs ReviewPublic

Authored by vlorentz on Jul 18 2022, 2:12 PM.

Details

Summary

Hash dicts are now prefered by swh-objstorage, in order to support
individual hash collisions.

This commit includes some minor fixture refactoring, to allow tests to
mock methods of the backend class even in proxy tests.

Test Plan

Depends on D8122

Diff Detail

Repository
rDSTO Storage manager
Branch
objid
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30436
Build 47579: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47578: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D8127 (id=29359)

Could not rebase; Attempt merge onto fbe3803820...

Updating fbe38038..f75ffe40
Fast-forward
 swh/storage/cassandra/cql.py                  | 12 ++++---
 swh/storage/cassandra/storage.py              | 29 ++++++++-------
 swh/storage/in_memory.py                      |  2 +-
 swh/storage/interface.py                      | 21 ++++++++---
 swh/storage/objstorage.py                     | 32 +++++++++++++----
 swh/storage/postgresql/storage.py             | 13 ++++---
 swh/storage/proxies/filter.py                 |  6 ++--
 swh/storage/pytest_plugin.py                  | 13 ++++++-
 swh/storage/tests/storage_tests.py            | 52 +++++++++++++++++++++++++++
 swh/storage/tests/test_api_client.py          |  5 +++
 swh/storage/tests/test_cassandra_migration.py | 14 ++++----
 swh/storage/tests/test_tenacious.py           | 12 +++----
 12 files changed, 156 insertions(+), 55 deletions(-)
Changes applied before test
commit f75ffe400bebd778450597c9af688fd2e8aa72d1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 14:11:09 2022 +0200

    Call objstorage.content_get() with a HashDict instead of single hash
    
    Hash dicts are now prefered by swh-objstorage, in order to support
    individual hash collisions.
    
    This commit includes some minor fixture refactoring, to allow tests to
    mock methods of the backend class even in proxy tests.

commit 21b16053f3098f41f58f5a49c9ea68b4140cdb64
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 13:51:44 2022 +0200

    Replace Dict[str, bytes] with a TypedDict to represent dicts of hashes
    
    Unfortunately, this requires jumping through a few hoops (`cast()` and
    `# type: ignore`) as mypy does not allow `d[k]` when `d` is a typed dict
    and `k` is not a literal string.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1636/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1636/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 18 2022, 2:27 PM
Harbormaster failed remote builds in B30436: Diff 29359!

Build was aborted

Patch application report for D8127 (id=29359)

Could not rebase; Attempt merge onto 5335244fc1...

Auto-merging swh/storage/tests/test_cassandra_migration.py
Auto-merging swh/storage/tests/test_api_client.py
Auto-merging swh/storage/tests/storage_tests.py
Auto-merging swh/storage/cassandra/storage.py
Auto-merging swh/storage/cassandra/cql.py
Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py                  | 12 ++++---
 swh/storage/cassandra/storage.py              | 29 ++++++++-------
 swh/storage/in_memory.py                      |  2 +-
 swh/storage/interface.py                      | 21 ++++++++---
 swh/storage/objstorage.py                     | 32 +++++++++++++----
 swh/storage/postgresql/storage.py             | 13 ++++---
 swh/storage/proxies/filter.py                 |  6 ++--
 swh/storage/pytest_plugin.py                  | 13 ++++++-
 swh/storage/tests/storage_tests.py            | 52 +++++++++++++++++++++++++++
 swh/storage/tests/test_api_client.py          |  5 +++
 swh/storage/tests/test_cassandra_migration.py | 14 ++++----
 swh/storage/tests/test_tenacious.py           | 12 +++----
 12 files changed, 156 insertions(+), 55 deletions(-)
Changes applied before test
commit ecc1248dad413ed8ecc7c842243a7e933cae4ab1
Merge: 5335244f f75ffe40
Author: Jenkins user <jenkins@localhost>
Date:   Tue Aug 23 06:08:17 2022 +0000

    Merge branch 'diff-target' into HEAD

commit f75ffe400bebd778450597c9af688fd2e8aa72d1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 14:11:09 2022 +0200

    Call objstorage.content_get() with a HashDict instead of single hash
    
    Hash dicts are now prefered by swh-objstorage, in order to support
    individual hash collisions.
    
    This commit includes some minor fixture refactoring, to allow tests to
    mock methods of the backend class even in proxy tests.

commit 21b16053f3098f41f58f5a49c9ea68b4140cdb64
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 13:51:44 2022 +0200

    Replace Dict[str, bytes] with a TypedDict to represent dicts of hashes
    
    Unfortunately, this requires jumping through a few hoops (`cast()` and
    `# type: ignore`) as mypy does not allow `d[k]` when `d` is a typed dict
    and `k` is not a literal string.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1657/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1657/console