Page MenuHomeSoftware Heritage

Call objstorage.content_get() with a HashDict instead of single hash
Needs ReviewPublic

Authored by vlorentz on Mon, Jul 18, 2:12 PM.

Details

Summary

Hash dicts are now prefered by swh-objstorage, in order to support
individual hash collisions.

This commit includes some minor fixture refactoring, to allow tests to
mock methods of the backend class even in proxy tests.

Test Plan

Depends on D8122

Diff Detail

Repository
rDSTO Storage manager
Branch
objid
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30436
Build 47579: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47578: arc lint + arc unit

Unit TestsFailed

TimeTest
35 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f5995803080> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f598fa02d68>
41 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_duplicate
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f598ff8ecc0> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f59901ace10>
28 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_from_lazy_content
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f598f237c50> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f598fa02320>
37 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_objstorage_first
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f598ff8e4a8> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f598ec5eba8>
32 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_get_data_full_dict
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f5995803c18> swh_storage = <RemoteStorage url=mock://example.com/> swh_storage_backend = <swh.storage.in_memory.InMemoryStorage object at 0x7f598fdd0ac8>
View Full Test Results (88 Failed · 1,557 Passed · 44 Skipped)

Event Timeline

Build has FAILED

Patch application report for D8127 (id=29359)

Could not rebase; Attempt merge onto fbe3803820...

Updating fbe38038..f75ffe40
Fast-forward
 swh/storage/cassandra/cql.py                  | 12 ++++---
 swh/storage/cassandra/storage.py              | 29 ++++++++-------
 swh/storage/in_memory.py                      |  2 +-
 swh/storage/interface.py                      | 21 ++++++++---
 swh/storage/objstorage.py                     | 32 +++++++++++++----
 swh/storage/postgresql/storage.py             | 13 ++++---
 swh/storage/proxies/filter.py                 |  6 ++--
 swh/storage/pytest_plugin.py                  | 13 ++++++-
 swh/storage/tests/storage_tests.py            | 52 +++++++++++++++++++++++++++
 swh/storage/tests/test_api_client.py          |  5 +++
 swh/storage/tests/test_cassandra_migration.py | 14 ++++----
 swh/storage/tests/test_tenacious.py           | 12 +++----
 12 files changed, 156 insertions(+), 55 deletions(-)
Changes applied before test
commit f75ffe400bebd778450597c9af688fd2e8aa72d1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 14:11:09 2022 +0200

    Call objstorage.content_get() with a HashDict instead of single hash
    
    Hash dicts are now prefered by swh-objstorage, in order to support
    individual hash collisions.
    
    This commit includes some minor fixture refactoring, to allow tests to
    mock methods of the backend class even in proxy tests.

commit 21b16053f3098f41f58f5a49c9ea68b4140cdb64
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jul 18 13:51:44 2022 +0200

    Replace Dict[str, bytes] with a TypedDict to represent dicts of hashes
    
    Unfortunately, this requires jumping through a few hoops (`cast()` and
    `# type: ignore`) as mypy does not allow `d[k]` when `d` is a typed dict
    and `k` is not a literal string.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1636/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1636/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mon, Jul 18, 2:27 PM
Harbormaster failed remote builds in B30436: Diff 29359!