Page MenuHomeSoftware Heritage

Object storage (RedHat collaboration)Milestone
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

Workboard to organise the collaboration between RedHat and Software Heritage on the Software Heritage object storage project

Recent Activity

Mar 25 2022

bchauvet raised the priority of T3054: Scale out object storage design from Normal to High.
Mar 25 2022, 5:28 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task

Mar 23 2022

bchauvet added a project to T3054: Scale out object storage design: Roadmap 2022.
Mar 23 2022, 5:05 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task

Jan 25 2022

dachary closed T3634: Create swh-perfecthash module, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Jan 25 2022, 9:51 PM · Object storage (RedHat collaboration)

Jan 24 2022

dachary changed the status of T3531: IO throttling: benchmark, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3531: IO throttling: benchmark from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation, a subtask of T3532: IO throttling, as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)

Jan 22 2022

dachary changed the status of T3532: IO throttling from Open to Work in Progress.
Jan 22 2022, 4:14 PM · Object storage (RedHat collaboration)

Jan 21 2022

dachary added a comment to T3530: IO throttling: implementation.

https://github.com/mjpieters/aiolimiter is a better fit and may be used for copy/pasting

Jan 21 2022, 4:28 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://github.com/vutran1710/PyrateLimiter looks really good but is not for IO throttling

Jan 21 2022, 4:18 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://www.django-rest-framework.org/api-guide/throttling/ is already used by swh-web/swh/web/api/throttling.py but is targeting REST API throttling, not a good fit.

Jan 21 2022, 4:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)

Jan 10 2022

dachary added a comment to T3533: Winery backend: basic implementation.

Nevermind. I just realized that https://forge.softwareheritage.org/D6796 was in the "Accepted" state and has been for almost two weeks. :facepalm:

Jan 10 2022, 9:32 AM · Object storage (RedHat collaboration)
dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Jan 10 2022, 9:30 AM · Object storage (RedHat collaboration)

Jan 3 2022

dachary added a comment to T3533: Winery backend: basic implementation.

Happy new year ! Or, in French, "Bonne année deux mille vingt d'oeufs" (pun on words courtesy Easter-Eggs).

Jan 3 2022, 11:24 AM · Object storage (RedHat collaboration)

Dec 27 2021

dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Dec 27 2021, 1:46 PM · Object storage (RedHat collaboration)

Dec 22 2021

dachary added a comment to T3533: Winery backend: basic implementation.

I updated D6796 to add a high level description of the files, classes and logic. I also split the WineryObjStorage class into four classes in an attempt to clarify the separation between the read and the write logic. I will now refrain from doing anything else: it can be difficult on the reviewers to follow a moving target :-D

Dec 22 2021, 2:11 PM · Object storage (RedHat collaboration)
dachary updated subscribers of T3533: Winery backend: basic implementation.

@olasd @vlorentz @vsellier @ardumont I'd be happy to start polishing the implementation after your first wave of review (thanks a lot for the time you spent on this, it is much appreciated 👍 ). I'm however concerned that it might make your work difficult because of the magnitude of the patch. An alternative would be to merge this first implementation, as long as you feel confident it won't break anything when deployed. It should not because it won't be instantiated. That would allow me to propose patches that are smaller and easier to review for:

Dec 22 2021, 1:04 AM · Object storage (RedHat collaboration)

Dec 20 2021

dachary updated subscribers of T3533: Winery backend: basic implementation.

@douardda will you have time for reviewing this week? I stand ready to address your comments. I'll stop working on that for now so that it is not a moving target.

Dec 20 2021, 9:21 AM · Object storage (RedHat collaboration)

Dec 14 2021

dachary added a revision to T3533: Winery backend: basic implementation: D6837: allocate 5012 as the winery service port.
Dec 14 2021, 5:34 PM · Object storage (RedHat collaboration)
dachary added a revision to T3533: Winery backend: basic implementation: D6834: docker: add the swh-winery-db and swh-winery services.
Dec 14 2021, 3:49 PM · Object storage (RedHat collaboration)
dachary updated the task description for T3533: Winery backend: basic implementation.
Dec 14 2021, 2:19 PM · Object storage (RedHat collaboration)

Dec 13 2021

dachary updated the task description for T3532: IO throttling.
Dec 13 2021, 2:26 PM · Object storage (RedHat collaboration)

Dec 12 2021

dachary closed T3524: Add winery backend: create the Ceph cluster as Resolved.

Duplicate of T3533

Dec 12 2021, 5:19 AM · Object storage (RedHat collaboration)

Dec 11 2021

dachary added a comment to T3533: Winery backend: basic implementation.

I think D6796 is ready for review and should be merged. It is not final and won't run in production but it is a working implementation of the object storage. From there I intend to setup a benchmarking environment on grid5000. Optimizations will be implemented to improve the benchmark results where it matters the most. The benchmarks that were run during the summer already provided insights which motivated the implementation of the shard packer in C.

Dec 11 2021, 10:36 PM · Object storage (RedHat collaboration)
dachary renamed T3533: Winery backend: basic implementation from Winery backend: implementation to Winery backend: basic implementation.
Dec 11 2021, 10:20 PM · Object storage (RedHat collaboration)

Dec 8 2021

dachary added a comment to T3533: Winery backend: basic implementation.

D6796 is a draft implementation that is not ready for review but can be reviewed anyways in case someone is curious.

Dec 8 2021, 5:28 PM · Object storage (RedHat collaboration)
dachary added a revision to T3533: Winery backend: basic implementation: D6796: winery: basic implementation of the backend.
Dec 8 2021, 5:21 PM · Object storage (RedHat collaboration)
dachary closed T3104: Persistent readonly perfect hash table, a subtask of T3054: Scale out object storage design, as Resolved.
Dec 8 2021, 5:07 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
dachary closed T3104: Persistent readonly perfect hash table as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3521: Persistent readonly perfect hash table: benchmarks as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3521: Persistent readonly perfect hash table: benchmarks, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3520: Persistent readonly perfect hash table: implementation as Resolved.
Dec 8 2021, 5:06 PM · Object storage (RedHat collaboration)
dachary closed T3520: Persistent readonly perfect hash table: implementation, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Dec 8 2021, 5:06 PM · Object storage (RedHat collaboration)

Nov 15 2021

dachary added a comment to T3327: Hardware architecture for the object storage.

Alternative hardware bill of material:

Nov 15 2021, 5:49 PM · Object storage (RedHat collaboration)

Nov 3 2021

olasd moved T3327: Hardware architecture for the object storage from Backlog to 2021-W44-45 on the Object storage (RedHat collaboration) board.
Nov 3 2021, 4:14 PM · Object storage (RedHat collaboration)
olasd moved T3532: IO throttling from Backlog to 2021-W44-45 on the Object storage (RedHat collaboration) board.
Nov 3 2021, 4:09 PM · Object storage (RedHat collaboration)

Nov 2 2021

olasd moved T3054: Scale out object storage design from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:22 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
olasd moved T3056: Ceph as an object storage from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:22 PM · Object storage (RedHat collaboration)
olasd moved T3104: Persistent readonly perfect hash table from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3249: Deleting and erasing an object from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3520: Persistent readonly perfect hash table: implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3521: Persistent readonly perfect hash table: benchmarks from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3524: Add winery backend: create the Ceph cluster from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3530: IO throttling: implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3531: IO throttling: benchmark from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3532: IO throttling from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3533: Winery backend: basic implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd edited projects for T3327: Hardware architecture for the object storage, added: Object storage (RedHat collaboration); removed Object storage.
Nov 2 2021, 4:20 PM · Object storage (RedHat collaboration)