Page MenuHomeSoftware Heritage
Differential D8029

Start introducing composite ObjId in the interface
ClosedPublic
Actions

Authored by vlorentz on Jun 22 2022, 5:19 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4403: Update objstorage interface to return dicts of hashes instead of single sha1
T4402: Pass dict of hashes instead of single sha1 to objstorage.get()
T2309: Add support for other hash algo than sha1 in current objstorage implementation
Commits
rDOBJS667cb87b9367: Start introducing composite ObjId in the interface
Summary

For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.

This changes the network protocol used by list_content from a
concatenation of fixed-length hashes to a stream of msgpacked
dictionaries.

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jun 22 2022, 5:19 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 22 2022, 5:19 PM
vlorentz updated this revision to Diff 28914.Jun 22 2022, 5:21 PM

remove commented-out code

swh-public-ci added a comment.Jun 22 2022, 5:22 PM

Build has FAILED

Patch application report for D8029 (id=28913)

Could not rebase; Attempt merge onto 847285a28d...

Updating 847285a..3bc1985
Fast-forward
 mypy.ini                                           |  3 +
 requirements.txt                                   |  1 +
 swh/objstorage/api/client.py                       | 21 ++++---
 swh/objstorage/api/server.py                       |  8 ++-
 swh/objstorage/backends/azure.py                   |  4 +-
 swh/objstorage/backends/generator.py               | 15 +++--
 swh/objstorage/backends/http.py                    |  6 +-
 swh/objstorage/backends/in_memory.py               | 10 ++--
 swh/objstorage/backends/libcloud.py                |  7 +--
 swh/objstorage/backends/pathslicing.py             | 66 +++++++---------------
 swh/objstorage/backends/seaweedfs/objstorage.py    |  6 +-
 swh/objstorage/backends/winery/objstorage.py       |  8 +--
 swh/objstorage/interface.py                        | 47 +++++++--------
 swh/objstorage/multiplexer/filter/filter.py        |  8 +--
 swh/objstorage/multiplexer/filter/id_filter.py     |  9 +--
 .../multiplexer/multiplexer_objstorage.py          | 24 +-------
 swh/objstorage/objstorage.py                       | 25 +++++---
 swh/objstorage/tests/objstorage_testing.py         |  5 +-
 swh/objstorage/tests/test_multiplexer_filter.py    | 19 -------
 .../tests/test_objstorage_multiplexer.py           |  7 ---
 .../tests/test_objstorage_pathslicing.py           |  7 ---
 .../tests/test_objstorage_random_generator.py      |  8 ++-
 swh/objstorage/tests/test_objstorage_winery.py     | 13 ++---
 23 files changed, 132 insertions(+), 195 deletions(-)
Changes applied before test
commit 3bc19851a1600520e58edb66fba6a440a4b189a7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 17:18:02 2022 +0200

    Start introducing composite ObjId in the interface
    
    For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.
    
    This changes the network protocol used by `list_content` from a
    concatenation of fixed-length hashes to a stream of msgpacked
    dictionaries.

commit 3491bf7eb24c7aaed2eabdef664787cb6a1aabdd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:56:39 2022 +0200

    Make WineryWriter.add return None
    
    Like WineryObjStorage and other ObjStorage backends.

commit 0fd9cefc33d50b1a345a3c965b7f63d7bcf58d61
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:18:47 2022 +0200

    Remove get_random()
    
    It is not used anywhere (and swh-storage provides content_get_random() that
    feels the same need), and cloud backends do not implement it anyway.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/163/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/163/console

Harbormaster failed remote builds in B30007: Diff 28913!Jun 22 2022, 5:22 PM
swh-public-ci added a comment.Jun 22 2022, 5:24 PM

Build has FAILED

Patch application report for D8029 (id=28914)

Could not rebase; Attempt merge onto 847285a28d...

Updating 847285a..4246996
Fast-forward
 mypy.ini                                           |  3 +
 requirements.txt                                   |  1 +
 swh/objstorage/api/client.py                       | 21 ++++---
 swh/objstorage/api/server.py                       |  8 ++-
 swh/objstorage/backends/azure.py                   |  4 +-
 swh/objstorage/backends/generator.py               | 15 +++--
 swh/objstorage/backends/http.py                    |  6 +-
 swh/objstorage/backends/in_memory.py               | 10 ++--
 swh/objstorage/backends/libcloud.py                |  7 +--
 swh/objstorage/backends/pathslicing.py             | 66 +++++++---------------
 swh/objstorage/backends/seaweedfs/objstorage.py    |  6 +-
 swh/objstorage/backends/winery/objstorage.py       |  8 +--
 swh/objstorage/interface.py                        | 46 ++++++---------
 swh/objstorage/multiplexer/filter/filter.py        |  8 +--
 swh/objstorage/multiplexer/filter/id_filter.py     |  9 +--
 .../multiplexer/multiplexer_objstorage.py          | 24 +-------
 swh/objstorage/objstorage.py                       | 25 +++++---
 swh/objstorage/tests/objstorage_testing.py         |  5 +-
 swh/objstorage/tests/test_multiplexer_filter.py    | 19 -------
 .../tests/test_objstorage_multiplexer.py           |  7 ---
 .../tests/test_objstorage_pathslicing.py           |  7 ---
 .../tests/test_objstorage_random_generator.py      |  8 ++-
 swh/objstorage/tests/test_objstorage_winery.py     | 13 ++---
 23 files changed, 129 insertions(+), 197 deletions(-)
Changes applied before test
commit 42469967ceb2ea2ea4866c3d4caed67ab1d1c0bd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 17:18:02 2022 +0200

    Start introducing composite ObjId in the interface
    
    For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.
    
    This changes the network protocol used by `list_content` from a
    concatenation of fixed-length hashes to a stream of msgpacked
    dictionaries.

commit 3491bf7eb24c7aaed2eabdef664787cb6a1aabdd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:56:39 2022 +0200

    Make WineryWriter.add return None
    
    Like WineryObjStorage and other ObjStorage backends.

commit 0fd9cefc33d50b1a345a3c965b7f63d7bcf58d61
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:18:47 2022 +0200

    Remove get_random()
    
    It is not used anywhere (and swh-storage provides content_get_random() that
    feels the same need), and cloud backends do not implement it anyway.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/164/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/164/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 22 2022, 5:24 PM
Harbormaster failed remote builds in B30008: Diff 28914!
vlorentz updated this revision to Diff 28925.Jun 23 2022, 3:12 PM

fix types + rebase

swh-public-ci added a comment.Jun 23 2022, 3:16 PM

Build is green

Patch application report for D8029 (id=28925)

Could not rebase; Attempt merge onto d7f4daa242...

Updating d7f4daa..91c6308
Fast-forward
 mypy.ini                                           |  3 +
 requirements.txt                                   |  1 +
 swh/objstorage/api/client.py                       | 21 ++++---
 swh/objstorage/api/server.py                       |  8 ++-
 swh/objstorage/backends/azure.py                   |  6 +-
 swh/objstorage/backends/generator.py               | 15 +++--
 swh/objstorage/backends/http.py                    |  8 +--
 swh/objstorage/backends/in_memory.py               | 12 ++--
 swh/objstorage/backends/libcloud.py                |  9 ++-
 swh/objstorage/backends/pathslicing.py             | 68 +++++++---------------
 swh/objstorage/backends/seaweedfs/objstorage.py    |  8 +--
 swh/objstorage/backends/winery/objstorage.py       |  8 +--
 swh/objstorage/interface.py                        | 46 +++++----------
 swh/objstorage/multiplexer/filter/filter.py        |  8 +--
 swh/objstorage/multiplexer/filter/id_filter.py     |  9 +--
 .../multiplexer/multiplexer_objstorage.py          | 26 ++-------
 swh/objstorage/objstorage.py                       | 25 +++++---
 swh/objstorage/tests/objstorage_testing.py         |  5 +-
 swh/objstorage/tests/test_multiplexer_filter.py    | 19 ------
 .../tests/test_objstorage_multiplexer.py           |  7 ---
 .../tests/test_objstorage_pathslicing.py           |  7 ---
 .../tests/test_objstorage_random_generator.py      |  8 ++-
 swh/objstorage/tests/test_objstorage_winery.py     | 13 ++---
 23 files changed, 136 insertions(+), 204 deletions(-)
Changes applied before test
commit 91c630852ab8204edcd068987cdaa0c23bfd9cff
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 17:18:02 2022 +0200

    Start introducing composite ObjId in the interface
    
    For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.
    
    This changes the network protocol used by `list_content` from a
    concatenation of fixed-length hashes to a stream of msgpacked
    dictionaries.

commit e712f28684a1a17233a700dd22606b594a3b617b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:56:39 2022 +0200

    Make WineryWriter.add return None
    
    Like WineryObjStorage and other ObjStorage backends.

commit 2caa05e869c952a3e8c4f3418d41ea17955f5447
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 16:18:47 2022 +0200

    Remove get_random()
    
    It is not used anywhere (and swh-storage provides content_get_random() that
    feels the same need), and cloud backends do not implement it anyway.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/167/ for more details.

Harbormaster completed remote builds in B30018: Diff 28925.Jun 23 2022, 3:16 PM
vlorentz requested review of this revision.Jun 23 2022, 3:16 PM
vlorentz added inline comments.Jun 23 2022, 3:23 PM
swh/objstorage/api/server.py
98–109

this code is actually covered, but it runs in a separate process

ardumont accepted this revision.Jun 24 2022, 11:36 AM
ardumont added a subscriber: ardumont.

lgtm, couple of remarks inline.

swh/objstorage/api/client.py
54–55
swh/objstorage/interface.py
77

no need, it's already typed in the signature.

swh/objstorage/objstorage.py
19

Please add a docstring so one can immediately see why we use a specific implementation over the default swh.model.hashutil.bytes_to_hex.

This revision is now accepted and ready to land.Jun 24 2022, 11:36 AM
bchauvet added a task: T2309: Add support for other hash algo than sha1 in current objstorage implementation.Jun 27 2022, 2:35 PM
vlorentz updated this revision to Diff 28944.Jun 27 2022, 3:09 PM
vlorentz marked 3 inline comments as done.

apply comments

vlorentz added inline comments.Jun 27 2022, 3:11 PM
swh/objstorage/interface.py
77

it's also incorrect; looks like I accidentally reverted a previous diff.

swh-public-ci added a comment.Jun 27 2022, 3:12 PM

Build is green

Patch application report for D8029 (id=28944)

Rebasing onto e712f28684...

Current branch diff-target is up to date.
Changes applied before test
commit d5974f38ea1c5f56f83c98ae5fd92e0c869e9e3f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 17:18:02 2022 +0200

    Start introducing composite ObjId in the interface
    
    For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.
    
    This changes the network protocol used by `list_content` from a
    concatenation of fixed-length hashes to a stream of msgpacked
    dictionaries.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/168/ for more details.

Harbormaster completed remote builds in B30035: Diff 28944.Jun 27 2022, 3:12 PM
douardda added a subscriber: douardda.Jul 1 2022, 10:22 AM
douardda added inline comments.
swh/objstorage/backends/seaweedfs/objstorage.py
129

shouldn't this be changed to objid_to_default_hex?

vlorentz updated this revision to Diff 29111.Jul 4 2022, 11:39 AM

seaweed: replace with objid_to_default_hex

swh-public-ci added a comment.Jul 4 2022, 11:42 AM

Build is green

Patch application report for D8029 (id=29111)

Rebasing onto e712f28684...

Current branch diff-target is up to date.
Changes applied before test
commit 667cb87b9367864523092c3e3f78ae50d4c24d71
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 22 17:18:02 2022 +0200

    Start introducing composite ObjId in the interface
    
    For now, any hash algo other than ID_HASH_ALGO ("sha1") is ignored.
    
    This changes the network protocol used by `list_content` from a
    concatenation of fixed-length hashes to a stream of msgpacked
    dictionaries.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/169/ for more details.

Harbormaster completed remote builds in B30202: Diff 29111.Jul 4 2022, 11:42 AM
Closed by commit rDOBJS667cb87b9367: Start introducing composite ObjId in the interface (authored by vlorentz). · Explain WhyJul 4 2022, 2:07 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDOBJS667cb87b9367: Start introducing composite ObjId in the interface.
vlorentz added a task: T4402: Pass dict of hashes instead of single sha1 to objstorage.get().Jul 19 2022, 3:16 PM
vlorentz added a task: T4403: Update objstorage interface to return dicts of hashes instead of single sha1.Jul 19 2022, 3:19 PM

Revision Contents
Changeset List

PathSize
3 lines
1 line
swh/
objstorage/
api/
20 lines
8 lines
backends/
6 lines
15 lines
8 lines
12 lines
9 lines
35 lines
seaweedfs/
11 lines
21 lines
multiplexer/
filter/
5 lines
4 lines
8 lines
22 lines
tests/
5 lines
8 lines

Diff 29141

View Options

mypy.ini

Loading...
View Options

requirements.txt

Loading...
View Options

swh/objstorage/api/client.py

Loading...
View Options

swh/objstorage/api/server.py

Loading...
View Options

swh/objstorage/backends/azure.py

Loading...
View Options

swh/objstorage/backends/generator.py

Loading...
View Options

swh/objstorage/backends/http.py

Loading...
View Options

swh/objstorage/backends/in_memory.py

Loading...
View Options

swh/objstorage/backends/libcloud.py

Loading...
View Options

swh/objstorage/backends/pathslicing.py

Loading...
View Options

swh/objstorage/backends/seaweedfs/objstorage.py

Loading...
View Options

swh/objstorage/interface.py

Loading...
View Options

swh/objstorage/multiplexer/filter/filter.py

Loading...
View Options

swh/objstorage/multiplexer/filter/id_filter.py

Loading...
View Options

swh/objstorage/multiplexer/multiplexer_objstorage.py

Loading...
View Options

swh/objstorage/objstorage.py

Loading...
View Options

swh/objstorage/tests/objstorage_testing.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_random_generator.py

Loading...
About · Developer information · Legal