Page MenuHomeSoftware Heritage

libcloud: implement an object path prefix
ClosedPublic

Authored by olasd on Nov 7 2019, 4:29 PM.

Details

Summary

This allows to store objects alongside other data in a given S3 bucket, which is
what we do in our current Amazon S3 export.

Depends on D2235

Test Plan

tox

Diff Detail

Repository
rDOBJS Object storage
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8875
Build 12953: tox-on-jenkinsJenkins
Build 12952: arc lint + arc unit

Event Timeline

Use _object_path instead of hashutil in libcloud prefix test

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/objstorage/backends/libcloud.py
49–50

Should probably mention that the last slashes will be ignored and one will be added.

65

Set it to '' by default and you can drop most of the conditionals below

This revision is now accepted and ready to land.Nov 7 2019, 4:53 PM
olasd marked an inline comment as done.

Rebase + Update for comments

olasd marked an inline comment as done.Nov 7 2019, 5:08 PM
olasd added inline comments.
swh/objstorage/backends/libcloud.py
49–50

That's what I meant by slash-separated; I'll rephrase.

65

I don't know how the library copes with accessing objects with a leading '/' and I don't really want to have to find out ;)

This revision was landed with ongoing or failed builds.Nov 7 2019, 5:10 PM
This revision was automatically updated to reflect the committed changes.
olasd marked an inline comment as done.
swh/objstorage/backends/libcloud.py
65

No, I meant this:

self.path_prefix = ''
if path_prefix:
    self.path_prefix = path_prefix.rstrip('/') + '/'