Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 8 2023

gitlab-migration closed T3533: Winery backend: basic implementation as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3532: IO throttling as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3531: IO throttling: benchmark as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3531: IO throttling: benchmark, a subtask of T3532: IO throttling, as Migrated.
Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3327: Hardware architecture for the object storage as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3249: Deleting and erasing an object, a subtask of T3054: Scale out object storage design, as Migrated.
Jan 8 2023, 10:23 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration closed T3249: Deleting and erasing an object as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3056: Ceph as an object storage as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:22 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3056: Ceph as an object storage, a subtask of T3054: Scale out object storage design, as Migrated.
Jan 8 2023, 10:22 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration closed T3054: Scale out object storage design as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:22 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3530: IO throttling: implementation, a subtask of T3532: IO throttling, from Resolved to Migrated.
Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3530: IO throttling: implementation from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3524: Add winery backend: create the Ceph cluster from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3521: Persistent readonly perfect hash table: benchmarks, a subtask of T3104: Persistent readonly perfect hash table, from Resolved to Migrated.
Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3520: Persistent readonly perfect hash table: implementation, a subtask of T3104: Persistent readonly perfect hash table, from Resolved to Migrated.
Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3521: Persistent readonly perfect hash table: benchmarks from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3520: Persistent readonly perfect hash table: implementation from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:02 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3104: Persistent readonly perfect hash table, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 10:01 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3104: Persistent readonly perfect hash table from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:01 PM · Object storage (RedHat collaboration)
gitlab-migration closed T3105: LSM vs Hash Table / RocksDB vs Citus, a subtask of T3054: Scale out object storage design, as Migrated.
Jan 8 2023, 5:02 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3634: Create swh-perfecthash module, a subtask of T3104: Persistent readonly perfect hash table, from Resolved to Migrated.
Jan 8 2023, 4:35 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3519: Persistent readonly perfect hash table: CI and package, a subtask of T3104: Persistent readonly perfect hash table, from Wontfix to Migrated.
Jan 8 2023, 4:34 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3422: Running the benchmarks: August 6th, 2021, 9 days, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:34 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3421: Running the benchmarks: July 16th, 2 days, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:34 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3210: Ceph Quincy CDS & immutable objects, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:34 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3186: Ceph Sepia lab for performance testing, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:34 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3149: Benchmark software for the object storage, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3107: Scale out object storage explorations, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3108: Grid5000 for benchmarking, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3106: TCO of a production ready Ambry vs implementing the design, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3088: Object storage index with redis, a subtask of T3054: Scale out object storage design, from Invalid to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3068: Using Sorted String Tables as a file format, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3066: Using RocksDB SST as a file format, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3065: Using git to store objects, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3064: Using ambry to store objects, a subtask of T3054: Scale out object storage design, from Invalid to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3057: Using seaweedfs to store objects, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3055: Ceph and immutable & append only storage, a subtask of T3056: Ceph as an object storage, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3052: Reducing Ceph bluestore_min_alloc_size from 64K to 4K, a subtask of T3056: Ceph as an object storage, from Invalid to Migrated.
Jan 8 2023, 4:33 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3051: Using EOS to store objects, a subtask of T3054: Scale out object storage design, from Resolved to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3050: Using libcephsqlite to store objects, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
gitlab-migration changed the status of T3049: Benchmarking an RBD based object container, a subtask of T3056: Ceph as an object storage, from Invalid to Migrated.
Jan 8 2023, 4:33 PM · Object storage (RedHat collaboration)
gitlab-migration changed the status of T3048: Using a custom Sorted String Table format, a subtask of T3054: Scale out object storage design, from Wontfix to Migrated.
Jan 8 2023, 4:33 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task

Mar 25 2022

bchauvet raised the priority of T3054: Scale out object storage design from Normal to High.
Mar 25 2022, 5:28 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task

Mar 23 2022

bchauvet added a project to T3054: Scale out object storage design: Roadmap 2022.
Mar 23 2022, 5:05 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task

Jan 25 2022

dachary closed T3634: Create swh-perfecthash module, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Jan 25 2022, 9:51 PM · Object storage (RedHat collaboration)

Jan 24 2022

dachary changed the status of T3531: IO throttling: benchmark, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3531: IO throttling: benchmark from Open to Work in Progress.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation, a subtask of T3532: IO throttling, as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)
dachary closed T3530: IO throttling: implementation as Resolved.
Jan 24 2022, 3:06 PM · Object storage (RedHat collaboration)

Jan 22 2022

dachary changed the status of T3532: IO throttling from Open to Work in Progress.
Jan 22 2022, 4:14 PM · Object storage (RedHat collaboration)

Jan 21 2022

dachary added a comment to T3530: IO throttling: implementation.

https://github.com/mjpieters/aiolimiter is a better fit and may be used for copy/pasting

Jan 21 2022, 4:28 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://github.com/vutran1710/PyrateLimiter looks really good but is not for IO throttling

Jan 21 2022, 4:18 PM · Object storage (RedHat collaboration)
dachary added a comment to T3530: IO throttling: implementation.

https://www.django-rest-framework.org/api-guide/throttling/ is already used by swh-web/swh/web/api/throttling.py but is targeting REST API throttling, not a good fit.

Jan 21 2022, 4:06 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation, a subtask of T3532: IO throttling, from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)
dachary changed the status of T3530: IO throttling: implementation from Open to Work in Progress.
Jan 21 2022, 3:53 PM · Object storage (RedHat collaboration)

Jan 10 2022

dachary added a comment to T3533: Winery backend: basic implementation.

Nevermind. I just realized that https://forge.softwareheritage.org/D6796 was in the "Accepted" state and has been for almost two weeks. :facepalm:

Jan 10 2022, 9:32 AM · Object storage (RedHat collaboration)
dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Jan 10 2022, 9:30 AM · Object storage (RedHat collaboration)

Jan 3 2022

dachary added a comment to T3533: Winery backend: basic implementation.

Happy new year ! Or, in French, "Bonne année deux mille vingt d'oeufs" (pun on words courtesy Easter-Eggs).

Jan 3 2022, 11:24 AM · Object storage (RedHat collaboration)

Dec 27 2021

dachary added a comment to T3533: Winery backend: basic implementation.

Gentle ping?

Dec 27 2021, 1:46 PM · Object storage (RedHat collaboration)

Dec 22 2021

dachary added a comment to T3533: Winery backend: basic implementation.

I updated D6796 to add a high level description of the files, classes and logic. I also split the WineryObjStorage class into four classes in an attempt to clarify the separation between the read and the write logic. I will now refrain from doing anything else: it can be difficult on the reviewers to follow a moving target :-D

Dec 22 2021, 2:11 PM · Object storage (RedHat collaboration)
dachary updated subscribers of T3533: Winery backend: basic implementation.

@olasd @vlorentz @vsellier @ardumont I'd be happy to start polishing the implementation after your first wave of review (thanks a lot for the time you spent on this, it is much appreciated 👍 ). I'm however concerned that it might make your work difficult because of the magnitude of the patch. An alternative would be to merge this first implementation, as long as you feel confident it won't break anything when deployed. It should not because it won't be instantiated. That would allow me to propose patches that are smaller and easier to review for:

Dec 22 2021, 1:04 AM · Object storage (RedHat collaboration)

Dec 20 2021

dachary updated subscribers of T3533: Winery backend: basic implementation.

@douardda will you have time for reviewing this week? I stand ready to address your comments. I'll stop working on that for now so that it is not a moving target.

Dec 20 2021, 9:21 AM · Object storage (RedHat collaboration)

Dec 14 2021

dachary added a revision to T3533: Winery backend: basic implementation: D6837: allocate 5012 as the winery service port.
Dec 14 2021, 5:34 PM · Object storage (RedHat collaboration)
dachary added a revision to T3533: Winery backend: basic implementation: D6834: docker: add the swh-winery-db and swh-winery services.
Dec 14 2021, 3:49 PM · Object storage (RedHat collaboration)
dachary updated the task description for T3533: Winery backend: basic implementation.
Dec 14 2021, 2:19 PM · Object storage (RedHat collaboration)

Dec 13 2021

dachary updated the task description for T3532: IO throttling.
Dec 13 2021, 2:26 PM · Object storage (RedHat collaboration)

Dec 12 2021

dachary closed T3524: Add winery backend: create the Ceph cluster as Resolved.

Duplicate of T3533

Dec 12 2021, 5:19 AM · Object storage (RedHat collaboration)

Dec 11 2021

dachary added a comment to T3533: Winery backend: basic implementation.

I think D6796 is ready for review and should be merged. It is not final and won't run in production but it is a working implementation of the object storage. From there I intend to setup a benchmarking environment on grid5000. Optimizations will be implemented to improve the benchmark results where it matters the most. The benchmarks that were run during the summer already provided insights which motivated the implementation of the shard packer in C.

Dec 11 2021, 10:36 PM · Object storage (RedHat collaboration)
dachary renamed T3533: Winery backend: basic implementation from Winery backend: implementation to Winery backend: basic implementation.
Dec 11 2021, 10:20 PM · Object storage (RedHat collaboration)

Dec 8 2021

dachary added a comment to T3533: Winery backend: basic implementation.

D6796 is a draft implementation that is not ready for review but can be reviewed anyways in case someone is curious.

Dec 8 2021, 5:28 PM · Object storage (RedHat collaboration)
dachary added a revision to T3533: Winery backend: basic implementation: D6796: winery: basic implementation of the backend.
Dec 8 2021, 5:21 PM · Object storage (RedHat collaboration)
dachary closed T3104: Persistent readonly perfect hash table, a subtask of T3054: Scale out object storage design, as Resolved.
Dec 8 2021, 5:07 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
dachary closed T3104: Persistent readonly perfect hash table as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3521: Persistent readonly perfect hash table: benchmarks as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3521: Persistent readonly perfect hash table: benchmarks, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Dec 8 2021, 5:07 PM · Object storage (RedHat collaboration)
dachary closed T3520: Persistent readonly perfect hash table: implementation as Resolved.
Dec 8 2021, 5:06 PM · Object storage (RedHat collaboration)
dachary closed T3520: Persistent readonly perfect hash table: implementation, a subtask of T3104: Persistent readonly perfect hash table, as Resolved.
Dec 8 2021, 5:06 PM · Object storage (RedHat collaboration)

Nov 15 2021

dachary added a comment to T3327: Hardware architecture for the object storage.

Alternative hardware bill of material:

Nov 15 2021, 5:49 PM · Object storage (RedHat collaboration)

Nov 3 2021

olasd moved T3327: Hardware architecture for the object storage from Backlog to 2021-W44-45 on the Object storage (RedHat collaboration) board.
Nov 3 2021, 4:14 PM · Object storage (RedHat collaboration)
olasd moved T3532: IO throttling from Backlog to 2021-W44-45 on the Object storage (RedHat collaboration) board.
Nov 3 2021, 4:09 PM · Object storage (RedHat collaboration)

Nov 2 2021

olasd moved T3054: Scale out object storage design from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:22 PM · Roadmap 2022, Object storage (RedHat collaboration), Roadmap 2021, meta-task
olasd moved T3056: Ceph as an object storage from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:22 PM · Object storage (RedHat collaboration)
olasd moved T3104: Persistent readonly perfect hash table from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3249: Deleting and erasing an object from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3520: Persistent readonly perfect hash table: implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3521: Persistent readonly perfect hash table: benchmarks from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3524: Add winery backend: create the Ceph cluster from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3530: IO throttling: implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3531: IO throttling: benchmark from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3532: IO throttling from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd moved T3533: Winery backend: basic implementation from Backlog to RedHat collaboration on the Object storage board.
Nov 2 2021, 4:21 PM · Object storage (RedHat collaboration)
olasd edited projects for T3327: Hardware architecture for the object storage, added: Object storage (RedHat collaboration); removed Object storage.
Nov 2 2021, 4:20 PM · Object storage (RedHat collaboration)
olasd created Object storage (RedHat collaboration).
Nov 2 2021, 4:16 PM