Page MenuHomeSoftware Heritage
Differential D49

Arbitrary slicing on PathSlicingObjStorage
ClosedPublic
Actions

Authored by qcampos on Jun 14 2016, 5:54 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T433: Refactor objstorage module
Commits
rDSTOC029a9512ddc5: storage: change objstorage initialization arguments
rDSTOCc7e1ed39a26a: objstorage.api: Change instantiation of the object storage
rDSTOC68abde3f243d: archiver: change objstorage initialization into the archiver
rDSTOCd048051ab19e: checker: change the objstorage initialization of the checker
rDSTOC746e0495b0c7: objstorage: PathslicingObjStorage now allow arbitrary slicing
rDSTOC252e49b15814: objstorage: move the constant ID_HASH_ALGO to ObjStorage
R65:68abde3f243d: archiver: change objstorage initialization into the archiver
R65:d048051ab19e: checker: change the objstorage initialization of the checker
R65:c7e1ed39a26a: objstorage.api: Change instantiation of the object storage
R65:029a9512ddc5: storage: change objstorage initialization arguments
R65:746e0495b0c7: objstorage: PathslicingObjStorage now allow arbitrary slicing
R65:252e49b15814: objstorage: move the constant ID_HASH_ALGO to ObjStorage
rDSTO68abde3f243d: archiver: change objstorage initialization into the archiver
rDSTOc7e1ed39a26a: objstorage.api: Change instantiation of the object storage
rDSTOd048051ab19e: checker: change the objstorage initialization of the checker
rDSTO746e0495b0c7: objstorage: PathslicingObjStorage now allow arbitrary slicing
rDSTO029a9512ddc5: storage: change objstorage initialization arguments
rDSTO252e49b15814: objstorage: move the constant ID_HASH_ALGO to ObjStorage
Summary

Allow the sha1 slicing of a content to be fully customizable.

For example, a content's sha1 : "abcdef1234567890" in a storage with slicing "0:2/0:5" will be stored at "root/ab/abcde/".

Also, make the required changes in the swh.storage package to follow those modifications.

Diff Detail

Repository
rDSTO Storage manager
Branch
T433
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 200
Build 302: Software Heritage Python tests
Build 301: arc lint + arc unit

Event Timeline

qcampos updated this revision to Diff 163.Jun 14 2016, 5:54 PM
qcampos added a task: T433: Refactor objstorage module.
qcampos retitled this revision from to Arbitrary slicing on PathSlicingObjStorage.
qcampos updated this object.
qcampos edited the test plan for this revision. (Show Details)
qcampos added a reviewer: olasd.
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 14 2016, 5:54 PM
qcampos updated this object.Jun 14 2016, 5:56 PM
qcampos edited edge metadata.
qcampos updated this revision to Diff 164.Jun 14 2016, 6:04 PM

Correct a docstring that was not up-to-date.

Herald edited edge metadata. · View Herald TranscriptJun 14 2016, 6:04 PM
qcampos updated this revision to Diff 170.Jun 15 2016, 4:54 PM
qcampos edited edge metadata.

Move a constant into the superclass.

Herald edited edge metadata. · View Herald TranscriptJun 15 2016, 4:54 PM
qcampos updated this revision to Diff 171.Jun 15 2016, 5:12 PM
qcampos edited edge metadata.

Add an default argument that was missing.

Herald edited edge metadata. · View Herald TranscriptJun 15 2016, 5:12 PM
olasd requested changes to this revision.Jun 15 2016, 5:37 PM
olasd edited edge metadata.

A few comments inline before merging :)

swh/storage/objstorage/objstorage_pathslicing.py
92

typo : 0:4 is only four characters long

121

You could instantiate the slice objects here (use slice(map(int)) instead of tuple(map(int))), and reuse them directly when constructing the path.

178–184

We should probably move that check at the instanciation of the storage rather than do it on each access: the length of an object id is constant.

185

hex_obj_id[bounds] for bounds in self.bounds instead of unpacking start, end and repacking them.

This revision now requires changes to proceed.Jun 15 2016, 5:37 PM
qcampos marked 3 inline comments as done.Jun 16 2016, 10:11 AM

Didn't knew I could create a slice item. Thanks !

swh/storage/objstorage/objstorage_pathslicing.py
178–184

Do we have a way, at instantiation, to know the size of a hash given the ID_HASH_ALGO algorithm without hard-coding it?

qcampos updated this revision to Diff 178.EditedJun 16 2016, 11:59 AM
qcampos edited edge metadata.

Correct a typo ;
and use a slice object instead of unpacking [start, stop] manually

olasd added inline comments.Jun 16 2016, 2:18 PM
swh/storage/objstorage/objstorage_pathslicing.py
178–184

Not really, no; we can add an ID_HASH_LENGTH variable next to ID_HASH_ALGO.

qcampos updated this revision to Diff 179.Jun 16 2016, 2:33 PM
qcampos marked 3 inline comments as done.
qcampos edited edge metadata.

Put the hash lenght test at initialization instead of doing it each access.

olasd added inline comments.Jun 16 2016, 2:39 PM
swh/storage/objstorage/objstorage.py
7

That should be 40 ! :)

qcampos updated this revision to Diff 180.Jun 16 2016, 2:51 PM

Correct the sha1 hexadecimal hash's length.

swh/storage/objstorage/objstorage.py
7

Woops! Thats better indeed.

qcampos marked 2 inline comments as done.Jun 16 2016, 2:51 PM
olasd accepted this revision.Jun 16 2016, 3:09 PM
olasd edited edge metadata.
This revision is now accepted and ready to land.Jun 16 2016, 3:09 PM
Closed by commit rDSTO252e49b15814: objstorage: move the constant ID_HASH_ALGO to ObjStorage (authored by qcampos). · Explain WhyJun 16 2016, 3:11 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
storage/
checker/
4 lines
objstorage/
api/
5 lines
5 lines
53 lines
4 lines
tests/
2 lines
4 lines
21 lines
6 lines

Diff 171

View Options

swh/storage/checker/checker.py

Loading...
View Options

swh/storage/objstorage/api/server.py

Loading...
View Options

swh/storage/objstorage/objstorage.py

Loading...
View Options

swh/storage/objstorage/objstorage_pathslicing.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/test_archiver.py

Loading...
View Options

swh/storage/tests/test_checker.py

Loading...
View Options

swh/storage/tests/test_objstorage_api.py

Loading...
View Options

swh/storage/tests/test_objstorage_pathslicing.py

Loading...
About · Developer information · Legal