Page MenuHomeSoftware Heritage

Implement the object storage multiplexer
ClosedPublic

Authored by qcampos on Jun 16 2016, 4:49 PM.

Details

Summary

The MultiplexerObjStorage can distribute objstorage operations to multiple implementations.

Also, the MultiplexerObjStorage will add filters to its storages, so when an operation will be required on a MultiplexerObjStorage, only storages with filters that allow this operation will be impacted.

Diff Detail

Repository
rDSTO Storage manager
Branch
T445
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 215
Build 321: Software Heritage Python tests
Build 320: arc lint + arc unit

Event Timeline

qcampos retitled this revision from to Implement the object storage multiplexer.
qcampos edited the test plan for this revision. (Show Details)
qcampos added a reviewer: olasd.
qcampos updated this object.
qcampos edited edge metadata.

Correct a mistake on a filter implementation.

Test coverage is not yet perfect (non-existant for the filters) but I made the diff to submit the existing code to review.

qcampos edited edge metadata.

Improve test coverage for the multiplexer and the filters.

ardumont added inline comments.
swh/storage/objstorage/multiplexer/filter/__init__.py
49

dictionary 's .get( method tries to get the key value (1st parameter) and defaults to the second parameter if not found.
(Don't hesitate to reply to me if i'm reading this wrong ^^)

88

biggest

swh/storage/objstorage/multiplexer/multiplexer_objstorage.py
20

there are no guarantees...

41

indicate

48

If the adding step...

49

adding step

swh/storage/tests/test_multiplexer_filter.py
341 ↗(On Diff #184)

Beware the print ^^

348 ↗(On Diff #184)

Here too.

swh/storage/tests/test_objstorage_multiplexer.py
55

Why do you add it using storage_v1 first and then the using the multiplexer storage after that?
Why don't you always use the multiplexer storage?

Sorry if it's obvious but i don't see it ^^

78

Shouldn't there also be an adding tryout step with the readonly storage and then when you access this new content, it's not there (since it's readonly)?

qcampos marked 9 inline comments as done.
qcampos edited edge metadata.

Correct some typo in docstrings & increase filters code coverage by adding tests for the regex filter.

swh/storage/objstorage/multiplexer/filter/__init__.py
49

What have I done? This is clearly not python. Indeed, that's simply get.

swh/storage/tests/test_objstorage_multiplexer.py
55

Here I'm cheating : as the multiplexer consider storage_v1 as a read-only storage, I wouldn't be able to add a content into it. So I bypass it and add the content directly into the storage without filter.

This way, I can test if the len of a multiplexer object storage detect all its storage and correctly return the number of files.

78

I've done it! The tests of the filters themselves are in test_multiplexer_filter.py. Here, I have no access to the read-only storage (only the initial storage, and the multiplexer).

swh/storage/tests/test_objstorage_multiplexer.py
55

Ah, yes...
I remember reading something like that in a docstring... Yes, in multiplexer_objstorage, add's docstring.
The content is considered added if all storages successfully add the content.
As the read-only won't be able to since it does nothing in that regard, then you cannot add it...
Thus the cheating step. ^^

78

Ok. Thanks!

ardumont abandoned this revision.
ardumont added a reviewer: qcampos.
qcampos reclaimed this revision.
qcampos edited reviewers, added: ardumont; removed: qcampos.
ardumont edited edge metadata.
This revision is now accepted and ready to land.Jun 21 2016, 2:37 PM
This revision was automatically updated to reflect the committed changes.