Page MenuHomeSoftware Heritage
Create Task
Maniphest T454

object storage backend that can read from/write to S3
Closed, MigratedEdits Locked
Actions

Description

To support mirroring our object storage on various public cloud offerings, we want to be able to read from/write to storage that speak the Amazon S3 API.

Ideally, we want to stay in the subset of the API that is spoken by several cloud providers, so some investigation is needed to identify the precise subset we need and want to target.

Revisions and Commits

rDOBJS Object storage
ClosedD94 Cloud-based objstorage using Apache Libcloud

Event Timeline

zack created this task.Jun 20 2016, 10:03 AM
zack updated the task description. (Show Details)
qcampos added a comment.Aug 3 2016, 5:21 PM

I've made some investigations : S3 does have GET and PUT operations on blobs that are referenced by a name (could be sha1). This is sufficient for our ObjStorage needs.

  • After research, Apache Libcloud seems to be a high level API that works for several clouds (Including Aws, Azure, Ceph, ...). It also have module for compute-type nodes. That might do what we need. However, libcloud is pretty high level and it may be difficult to create our own driver if we need to use an S3 API on a cloud that is not managed by the lib. Need more digging to be sure.
  • Second idea would be to look at boto (Note: Boto works with python3, and Boto3 seems to hide low-level features we may need to access a non-Amazon S3 service, so basic Boto may be the best), but it's only a S3 API.
  • I've also looked at the S3 protocol to see if making our own generic S3 driver would be possible. It seems the authentication protocol might be difficult and require sore work.
qcampos added a revision: D94: Cloud-based objstorage using Apache Libcloud.Aug 5 2016, 4:32 PM
zack removed qcampos as the assignee of this task.Sep 15 2017, 9:55 AM
zack added a subscriber: qcampos.
douardda closed this task as Resolved.Mar 12 2019, 9:48 AM
douardda claimed this task.
douardda added a subscriber: douardda.

Unless I'm wrong, this task can be closed now. There is a libcloud-based backend.

gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:19 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal