Page MenuHomeSoftware Heritage

dachary (Loïc Dachary)
User

Projects

User does not belong to any projects.

User Details

User Since
Jan 8 2021, 11:21 PM (13 w, 16 h)

Recent Activity

Wed, Apr 7

dachary added a comment to T3149: Benchmark software for the object storage.

The benchmark was moved to a temporary repository for convenience (easier than uploading here every time). https://git.easter-eggs.org/biceps/biceps

Wed, Apr 7, 6:25 PM · Object storage

Tue, Apr 6

dachary closed T3210: Ceph Quincy CDS & immutable objects as Resolved.
Tue, Apr 6, 11:33 PM · Object storage
dachary closed T3210: Ceph Quincy CDS & immutable objects, a subtask of T3054: Scale out object storage design, as Resolved.
Tue, Apr 6, 11:33 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Tue, Apr 6, 11:33 PM · Roadmap 2021, meta-task, Object storage
dachary added a comment to T3210: Ceph Quincy CDS & immutable objects.

Takeaways from the session:

Tue, Apr 6, 6:35 PM · Object storage
dachary updated the task description for T3210: Ceph Quincy CDS & immutable objects.
Tue, Apr 6, 1:46 PM · Object storage
dachary updated the task description for T3210: Ceph Quincy CDS & immutable objects.
Tue, Apr 6, 1:46 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3210: Ceph Quincy CDS & immutable objects.
Tue, Apr 6, 1:39 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3210: Ceph Quincy CDS & immutable objects: T3054: Scale out object storage design.
Tue, Apr 6, 1:39 PM · Object storage
dachary changed the status of T3210: Ceph Quincy CDS & immutable objects from Open to Work in Progress.
Tue, Apr 6, 1:33 PM · Object storage

Tue, Mar 30

dachary renamed T3186: Ceph Sepia lab for performance testing from Ceph Sepia lab for testing to Ceph Sepia lab for performance testing.
Tue, Mar 30, 10:14 AM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3186: Ceph Sepia lab for performance testing.
Tue, Mar 30, 10:13 AM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3186: Ceph Sepia lab for performance testing: T3054: Scale out object storage design.
Tue, Mar 30, 10:13 AM · Object storage
dachary changed the status of T3186: Ceph Sepia lab for performance testing from Open to Work in Progress.
Tue, Mar 30, 10:13 AM · Object storage

Fri, Mar 26

dachary updated the task description for T3054: Scale out object storage design.
Fri, Mar 26, 11:52 PM · Roadmap 2021, meta-task, Object storage

Thu, Mar 25

dachary updated the task description for T3054: Scale out object storage design.
Thu, Mar 25, 10:18 AM · Roadmap 2021, meta-task, Object storage

Wed, Mar 24

dachary added a comment to T3149: Benchmark software for the object storage.

Refactored the custer provsioning to use all available disks instead of the existing file system (using cephadm instead of a hand made ceph cluster).

Wed, Mar 24, 11:50 AM · Object storage

Tue, Mar 23

dachary added a comment to T3149: Benchmark software for the object storage.

The benchmark runs and it's not too complicated which is a relief. I'll cleanup the mess I made and move forward to finish writing the software.

Tue, Mar 23, 3:27 PM · Object storage
dachary added a comment to T3149: Benchmark software for the object storage.

The benchmarks are not fully functional but they produce a write load that matches the object storage design. They run (README.txt) via libvirt and are being tested on Grid5000 to ensure all the pieces are in place (i.e. does it actually work to reserve machines + provision them + run) before moving forward.

Tue, Mar 23, 3:03 PM · Object storage

Wed, Mar 17

dachary added a comment to T3057: Using seaweedfs to store objects.

Mail thread with Chris Lu on SeaweedFS use cases with 100+ billions objects.

Wed, Mar 17, 4:22 PM · Object storage
dachary updated the task description for T3054: Scale out object storage design.
Wed, Mar 17, 4:18 PM · Roadmap 2021, meta-task, Object storage
dachary added a subtask for T3054: Scale out object storage design: T3149: Benchmark software for the object storage.
Wed, Mar 17, 4:16 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3149: Benchmark software for the object storage: T3054: Scale out object storage design.
Wed, Mar 17, 4:16 PM · Object storage
dachary added a comment to T3149: Benchmark software for the object storage.

First draft for layer 0.

Wed, Mar 17, 4:16 PM · Object storage
dachary changed the status of T3149: Benchmark software for the object storage from Open to Work in Progress.
Wed, Mar 17, 4:15 PM · Object storage

Mon, Mar 15

dachary added a comment to T3054: Scale out object storage design.

Bookmarking https://leo-project.net/leofs/

Mon, Mar 15, 5:21 PM · Roadmap 2021, meta-task, Object storage

Mar 10 2021

dachary closed T3108: Grid5000 for benchmarking as Resolved.
Mar 10 2021, 9:10 PM · Object storage
dachary closed T3108: Grid5000 for benchmarking, a subtask of T3054: Scale out object storage design, as Resolved.
Mar 10 2021, 9:10 PM · Roadmap 2021, meta-task, Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

With a little help from the mattermost channel and after approval of the account, it was possible to boot a physical machine with a Debian GNU/Linux installed from scratch and get root access to it.

Mar 10 2021, 9:09 PM · Object storage
dachary updated the task description for T3108: Grid5000 for benchmarking.
Mar 10 2021, 5:41 PM · Object storage
dachary added a comment to T3054: Scale out object storage design.

Thanks for helping with the labelling @rdicosmo 👍

Mar 10 2021, 4:30 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 10 2021, 4:11 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 10 2021, 4:10 PM · Roadmap 2021, meta-task, Object storage
dachary closed T3106: TCO of a production ready Ambry vs implementing the design as Resolved.

Added a section about TCO in the design document.

Mar 10 2021, 9:17 AM · Object storage
dachary closed T3106: TCO of a production ready Ambry vs implementing the design, a subtask of T3054: Scale out object storage design, as Resolved.
Mar 10 2021, 9:17 AM · Roadmap 2021, meta-task, Object storage

Mar 9 2021

dachary added a comment to T3108: Grid5000 for benchmarking.

There is a mattermost channel dedicated to Grid5000 but one has to be invited to join, it is not open to the public.

Mar 9 2021, 10:48 PM · Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

Additional nvme drives for yeti should be something similar to https://www.samsung.com/semiconductor/ssd/enterprise-ssd/ but confirmation is needed to verify the machines actually have the required SFF-8639 to plug them in.

Mar 9 2021, 7:01 PM · Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

The account request was approved, I'll proceed with a minimal reservation to figure out how it is done.

Mar 9 2021, 6:34 PM · Object storage
dachary updated the task description for T3108: Grid5000 for benchmarking.
Mar 9 2021, 6:18 PM · Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

Thanks for the feedback. https://www.grid5000.fr/w/Grenoble:Hardware#yeti has 1.6TB nvme which seems better. It would be better to have a total of 4TB nvme available to get closer to the target global index size (i.e. 40 bytes 100 billions entries = 4TB). I'm told it is possible to donate hardware to Grid5000: if testing with the current configuration is not convincing enough, 4 more nvme pcie drives could be donated and they would be installed in the machines. No idea how much delay to expect but its good to know it is possible.

Mar 9 2021, 6:11 PM · Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

Looking at the available hardware, here is what could be used:

Mar 9 2021, 4:17 PM · Object storage
dachary added a comment to T3108: Grid5000 for benchmarking.

Followed the instructions at https://www.grid5000.fr/w/Grid5000:Get_an_account to get an account. Waiting for approval.

Mar 9 2021, 4:02 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3108: Grid5000 for benchmarking.
Mar 9 2021, 3:12 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3108: Grid5000 for benchmarking: T3054: Scale out object storage design.
Mar 9 2021, 3:12 PM · Object storage
dachary changed the status of T3108: Grid5000 for benchmarking from Open to Work in Progress.
Mar 9 2021, 3:12 PM · Object storage
dachary renamed T3105: LSM vs Hash Table / RocksDB vs Citus from LSM vs Hash Table / RocksDB vs Citrus to LSM vs Hash Table / RocksDB vs Citus.
Mar 9 2021, 3:00 PM · Object storage
dachary updated the task description for T3105: LSM vs Hash Table / RocksDB vs Citus.
Mar 9 2021, 3:00 PM · Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 9 2021, 1:26 PM · Roadmap 2021, meta-task, Object storage
dachary added a subtask for T3054: Scale out object storage design: T3107: Scale out object storage explorations.
Mar 9 2021, 1:09 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3107: Scale out object storage explorations: T3054: Scale out object storage design.
Mar 9 2021, 1:09 PM · Object storage
dachary closed T3107: Scale out object storage explorations as Resolved.
Mar 9 2021, 1:09 PM · Object storage

Mar 8 2021

dachary added a parent task for T3106: TCO of a production ready Ambry vs implementing the design: T3054: Scale out object storage design.
Mar 8 2021, 10:35 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3106: TCO of a production ready Ambry vs implementing the design.
Mar 8 2021, 10:35 PM · Roadmap 2021, meta-task, Object storage
dachary triaged T3106: TCO of a production ready Ambry vs implementing the design as Normal priority.
Mar 8 2021, 10:35 PM · Object storage
dachary added a parent task for T3105: LSM vs Hash Table / RocksDB vs Citus: T3054: Scale out object storage design.
Mar 8 2021, 10:29 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3105: LSM vs Hash Table / RocksDB vs Citus.
Mar 8 2021, 10:29 PM · Roadmap 2021, meta-task, Object storage
dachary triaged T3105: LSM vs Hash Table / RocksDB vs Citus as Normal priority.
Mar 8 2021, 10:29 PM · Object storage
dachary added a parent task for T3104: Using a custom Hash Table format: T3054: Scale out object storage design.
Mar 8 2021, 10:09 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3104: Using a custom Hash Table format.
Mar 8 2021, 10:09 PM · Roadmap 2021, meta-task, Object storage
dachary changed the status of T3104: Using a custom Hash Table format from Open to Work in Progress.
Mar 8 2021, 10:08 PM · Object storage
dachary added a comment to T3048: Using a custom Sorted String Table format.

https://en.wikipedia.org/wiki/Perfect_hash_function could be used to get O(1) instead of O(log(N))

Mar 8 2021, 9:53 PM · Object storage
dachary added a comment to T3054: Scale out object storage design.

Very interesting to see how this problem was presented & solved in the Hadoop ecosystem, thanks for the links.

Mar 8 2021, 10:06 AM · Roadmap 2021, meta-task, Object storage

Mar 5 2021

dachary closed T3088: Object storage index with redis, a subtask of T3054: Scale out object storage design, as Invalid.
Mar 5 2021, 8:32 AM · Roadmap 2021, meta-task, Object storage
dachary closed T3088: Object storage index with redis as Invalid.
Mar 5 2021, 8:32 AM · Object storage
dachary updated the task description for T3088: Object storage index with redis.
Mar 5 2021, 8:32 AM · Object storage
dachary updated the task description for T3088: Object storage index with redis.
Mar 5 2021, 8:27 AM · Object storage

Mar 4 2021

dachary updated the task description for T3054: Scale out object storage design.
Mar 4 2021, 11:23 PM · Roadmap 2021, meta-task, Object storage
dachary added a subtask for T3054: Scale out object storage design: T3088: Object storage index with redis.
Mar 4 2021, 10:45 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3088: Object storage index with redis: T3054: Scale out object storage design.
Mar 4 2021, 10:45 PM · Object storage
dachary changed the status of T3088: Object storage index with redis from Open to Work in Progress.
Mar 4 2021, 10:45 PM · Object storage

Mar 3 2021

dachary updated the task description for T3054: Scale out object storage design.
Mar 3 2021, 1:48 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 3 2021, 1:17 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 3 2021, 8:23 AM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 3 2021, 1:20 AM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 3 2021, 12:52 AM · Roadmap 2021, meta-task, Object storage

Mar 2 2021

dachary updated the task description for T3054: Scale out object storage design.
Mar 2 2021, 11:47 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 2 2021, 11:43 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 2 2021, 4:30 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 2 2021, 3:32 PM · Roadmap 2021, meta-task, Object storage

Mar 1 2021

dachary updated the task description for T3054: Scale out object storage design.
Mar 1 2021, 4:06 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 1 2021, 11:57 AM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Mar 1 2021, 11:55 AM · Roadmap 2021, meta-task, Object storage

Feb 24 2021

dachary added a comment to T3054: Scale out object storage design.

For the record the half baked benchmark script for the proposed designed I worked on today. To be continued!


Feb 24 2021, 6:16 PM · Roadmap 2021, meta-task, Object storage

Feb 23 2021

dachary updated the task description for T3048: Using a custom Sorted String Table format.
Feb 23 2021, 11:26 PM · Object storage
dachary renamed T3048: Using a custom Sorted String Table format from Using a custom Sorted String format to Using a custom Sorted String Table format.
Feb 23 2021, 11:24 PM · Object storage
dachary added a comment to T3048: Using a custom Sorted String Table format.

Reopening for benchmarking purposes because there does not seem to be anything ready to use T3068.

Feb 23 2021, 11:22 PM · Object storage
dachary reopened T3048: Using a custom Sorted String Table format, a subtask of T3054: Scale out object storage design, as Work in Progress.
Feb 23 2021, 11:21 PM · Roadmap 2021, meta-task, Object storage
dachary renamed T3048: Using a custom Sorted String Table format from Using a custom format for 1TB archive to Using a custom Sorted String format.
Feb 23 2021, 11:21 PM · Object storage
dachary updated the task description for T3068: Using Sorted String Tables as a file format.
Feb 23 2021, 1:53 PM · Object storage
dachary updated the task description for T3054: Scale out object storage design.
Feb 23 2021, 1:52 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Feb 23 2021, 1:51 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Feb 23 2021, 1:40 PM · Roadmap 2021, meta-task, Object storage
dachary updated the task description for T3054: Scale out object storage design.
Feb 23 2021, 1:16 PM · Roadmap 2021, meta-task, Object storage
dachary closed T3069: Using MZ as a file format as Invalid.
Feb 23 2021, 1:15 PM · Object storage
dachary added a comment to T3068: Using Sorted String Tables as a file format.

jumpDB is 100% python therefore less than ideal for CPU performance but for the purpose of benchmarking I/O and space usage it is conveniently ready to use.

Feb 23 2021, 12:15 PM · Object storage
dachary updated the task description for T3068: Using Sorted String Tables as a file format.
Feb 23 2021, 12:13 PM · Object storage
dachary updated the task description for T3068: Using Sorted String Tables as a file format.
Feb 23 2021, 12:06 PM · Object storage
dachary added a subtask for T3054: Scale out object storage design: T3068: Using Sorted String Tables as a file format.
Feb 23 2021, 12:04 PM · Roadmap 2021, meta-task, Object storage
dachary added a parent task for T3068: Using Sorted String Tables as a file format: T3054: Scale out object storage design.
Feb 23 2021, 12:04 PM · Object storage
dachary changed the status of T3068: Using Sorted String Tables as a file format from Open to Work in Progress.
Feb 23 2021, 12:04 PM · Object storage