Page MenuHomeSoftware Heritage

Add a seaweedfs backend
ClosedPublic

Authored by douardda on Mar 19 2019, 12:31 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Summary

an add support for the seaweedfs backend in get_objstorage()

Depends on D1275

Diff Detail

Branch
master
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 4743
Build 6321: arc lint + arc unit

Event Timeline

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/objstorage/__init__.py
55–59

There is no reason for this backend to raise an ImportError.

swh/objstorage/objstorage_weed.py
22

should probably remove that line

87–92

You can do it without getting the content by using an HEAD request. (I just checked, it works)

94–118

Incorrect, it does not list recursively, as far as I understand.

185–188

It's missing decompression

190–200

Duplicate content of __iter__ (with the same bug)

This revision now requires changes to proceed.Mar 19 2019, 2:25 PM
douardda added inline comments.
swh/objstorage/__init__.py
55–59

Indeed. This comes from a previous version of the diff in which I used to depends on an external package. Will fix that.

swh/objstorage/objstorage_weed.py
22

Same reason as above.

87–92

Good thinking.

94–118

why would we recurse? Everything is stored flat in there (for now at least). But yes, the len is incorrect as is. I'll try to find a better implementation, if any.

190–200

indeed

swh/objstorage/objstorage_weed.py
94–118

Everything is stored flat

Oh, right

douardda added inline comments.
swh/objstorage/objstorage_weed.py
22

In fact, aiohttp does generate a lot of INFO level logs, so this is the quick'n'dirty way to prevent that

What happened to get_stream?

It needs more attention, so I prefer postpone its implementation for now.

This revision is now accepted and ready to land.Mar 20 2019, 11:42 AM

closed by 167aabef25b10299dc2ebf82d0d4a1859b03683f