Page MenuHomeSoftware Heritage
Differential D1298

Add a seaweedfs backend
ClosedPublic
Actions

Authored by douardda on Mar 27 2019, 11:25 AM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDOBJS167aabef25b1: Add a seaweedfs backend
Summary

an add support for the seaweedfs backend in get_objstorage().
Note that streaming methods have not been implemented yet.

Depends on D1275

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Mar 27 2019, 11:25 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 27 2019, 11:25 AM
Harbormaster failed remote builds in B4852: Diff 4104!Mar 27 2019, 11:25 AM
douardda added a child revision: D1299: api/server: insert statsd probes for every API endpoint.Mar 27 2019, 11:25 AM
swh-public-ci added a subscriber: swh-public-ci.Mar 27 2019, 11:25 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/34/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/34/console

olasd requested changes to this revision.Mar 27 2019, 11:43 AM
olasd added a subscriber: olasd.

The top level documentation for both classes needs some love (or at least it'd be nicer if it wasn't empty :D)

The WeedFiler class does no real error checking at all which doesn't leave a great feeling.

No tests? :(

swh/objstorage/objstorage_weed.py
80–81

Looks like this shouldn't be hardcoded?

This revision now requires changes to proceed.Mar 27 2019, 11:43 AM
vlorentz added a subscriber: vlorentz.Mar 27 2019, 2:19 PM

Duplicate of D1276.

douardda updated this revision to Diff 4128.Mar 27 2019, 2:53 PM

Add tests and (consequently) few fixes

swh-public-ci added a comment.Mar 27 2019, 2:54 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/42/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/42/console

Harbormaster failed remote builds in B4880: Diff 4128!Mar 27 2019, 2:54 PM
douardda updated this revision to Diff 4131.Mar 27 2019, 3:03 PM

fix path handling

swh-public-ci added a comment.Mar 27 2019, 3:05 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/43/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/43/console

Harbormaster failed remote builds in B4885: Diff 4131!Mar 27 2019, 3:05 PM
douardda updated this revision to Diff 4135.Mar 27 2019, 3:09 PM

rebased

swh-public-ci added a comment.Mar 27 2019, 3:10 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/46/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/46/console

Harbormaster failed remote builds in B4889: Diff 4135!Mar 27 2019, 3:11 PM
olasd requested changes to this revision.Mar 27 2019, 4:23 PM

Sorry for going for another loop, but that's still missing error checking on get/put/delete methods of the WeedFiler: these methods will return truthy values even when the server responds (properly) with an error condition and has failed to comply.

This revision now requires changes to proceed.Mar 27 2019, 4:23 PM
douardda added a comment.Mar 27 2019, 4:37 PM
In D1298#27999, @olasd wrote:

Sorry for going for another loop

don't apologize. I'm the one that have been lazy!

douardda added a comment.EditedMar 28 2019, 9:57 AM
In D1298#27999, @olasd wrote:

Sorry for going for another loop, but that's still missing error checking on get/put/delete methods of the WeedFiler: these methods will return truthy values even when the server responds (properly) with an error condition and has failed to comply.

On the other side, the ObjStorage 'API' (as defined in swh/objstorage/objstorage.py) does not describe how errors should be handled. So I can add some error checking code in the WeedFiler class, but I'm not quite clear what to do with these errors on the ObjStorage class part...

So I am tempted to add XXX in this code for now.

douardda updated this revision to Diff 4156.Mar 28 2019, 10:27 AM

Add a big bad TODO

swh-public-ci added a comment.Mar 28 2019, 10:29 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/50/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tox/50/console

Harbormaster failed remote builds in B4913: Diff 4156!Mar 28 2019, 10:29 AM
olasd accepted this revision.Mar 28 2019, 10:42 AM
This revision is now accepted and ready to land.Mar 28 2019, 10:42 AM
This revision was landed with ongoing or failed builds.Mar 28 2019, 10:57 AM
Closed by commit rDOBJS167aabef25b1: Add a seaweedfs backend (authored by douardda). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
objstorage/
2 lines
202 lines
tests/
45 lines

Diff 4159

View Options

swh/objstorage/__init__.py

Loading...
View Options

swh/objstorage/objstorage_weed.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_seaweedfs.py

Loading...
About · Developer information · Legal