Page MenuHomeSoftware Heritage

libcloud: implement an object path prefix
ClosedPublic

Authored by olasd on Thu, Nov 7, 4:29 PM.

Details

Summary

This allows to store objects alongside other data in a given S3 bucket, which is
what we do in our current Amazon S3 export.

Depends on D2235

Test Plan

tox

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Thu, Nov 7, 4:29 PM
olasd updated this revision to Diff 7708.Thu, Nov 7, 4:34 PM

Rebase

olasd updated this revision to Diff 7709.Thu, Nov 7, 4:41 PM

Use _object_path instead of hashutil in libcloud prefix test

vlorentz accepted this revision.Thu, Nov 7, 4:53 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/objstorage/backends/libcloud.py
50–51

Should probably mention that the last slashes will be ignored and one will be added.

66

Set it to '' by default and you can drop most of the conditionals below

This revision is now accepted and ready to land.Thu, Nov 7, 4:53 PM
olasd updated this revision to Diff 7713.Thu, Nov 7, 5:08 PM
olasd marked an inline comment as done.

Rebase + Update for comments

olasd marked an inline comment as done.Thu, Nov 7, 5:08 PM
olasd added inline comments.
swh/objstorage/backends/libcloud.py
50–51

That's what I meant by slash-separated; I'll rephrase.

66

I don't know how the library copes with accessing objects with a leading '/' and I don't really want to have to find out ;)

This revision was landed with ongoing or failed builds.Thu, Nov 7, 5:10 PM
This revision was automatically updated to reflect the committed changes.
olasd marked an inline comment as done.
vlorentz added inline comments.Thu, Nov 7, 5:22 PM
swh/objstorage/backends/libcloud.py
66

No, I meant this:

self.path_prefix = ''
if path_prefix:
    self.path_prefix = path_prefix.rstrip('/') + '/'