A mail was sent to Patrick Donnelly to ask for his opinion on the matter.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Feb 15 2021
Feb 15 2021
dachary changed the status of T3050: Using libcephsqlite to store objects from Open to Work in Progress.
dachary closed T3014: Using an RBD image to store artifacts, a subtask of T3049: Benchmarking an RBD based object container, as Resolved.
This preliminary exploration is complete and moved to benchmarking to discover blockers.
dachary added a parent task for T3014: Using an RBD image to store artifacts: T3049: Benchmarking an RBD based object container.
Updated the description, even simpler.
Thanks for the comment. Let's keep just the SWHID then.
dachary changed the status of T3049: Benchmarking an RBD based object container from Open to Work in Progress.
dachary renamed T3048: Using a custom Sorted String Table format from Using custom format for 1TB archive to Using a custom format for 1TB archive.
dachary added a parent task for T3046: Using xz-file-format for 1TB archive: T3048: Using a custom Sorted String Table format.
dachary added a parent task for T3045: Using pixz for 1TB archives: T3048: Using a custom Sorted String Table format.
dachary changed the status of T3048: Using a custom Sorted String Table format from Open to Work in Progress.
Although simple and close to what is needed, Xz is not an exact match: the index would need to be maintained.
Xz format inadequate for long-term archiving
dachary changed the status of T3014: Using an RBD image to store artifacts from Open to Work in Progress.
The zstd format is tightly associated with the compression algorithm and is therefore more complex. It can however be a sequence of independently compressed content and could be used for the same purpose as xz.
The 7z format is more complex because it knows about files, directories etc. It is not not just a compressed data format.
dachary changed the status of T3046: Using xz-file-format for 1TB archive from Open to Work in Progress.
There are two blockers:
When extracting a single file (-x file) the in memory index is walked sequentially looking for the file.
The index is located at the end of the file.
The content of the archive is compressed as successive blocs of a given size.
The index is compressed as a single block of unlimited size.
Feb 14 2021
Feb 14 2021
In D398#24602, @douardda wrote:No idea whether if it's of some interest for our subject, but we may also have a look at openio
About Ceph RGW and the lack of packing https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AEMW6O7WVJFMUIX7QGI2KM7HKDSTNIYT/
https://github.com/vasi/pixz is a candidate for the 1TB archive content
For the record yesterday's IRC log
Feb 13 2021
Feb 13 2021
For the record, today's IRC log:
Feb 6 2021
Feb 6 2021
Benchmarking S3 in Ceph with COSBench could be interesting (the video is not yet available). In the past COSBench was difficult to use but maybe it improved. This is off-topic though, but I don't know where to write that down at the moment.
Feb 4 2021
Feb 4 2021
Feb 2 2021
Feb 2 2021
Feb 1 2021
Feb 1 2021
A trivial test case (attached) shows that an RBD image backed by a k=4,m=2 erasure coded pool (RAID6 equivalent) can store 4GB of data using 6GB of disk. The metadata overhead is small. It would be great if someone could repeat the test to make sure I did not accidentally obtained these results.
Jan 11 2021
Jan 11 2021
Thanks for the merge :-) It feels really good to see a commit, however simple, being merged on a Monday morning!
Jan 9 2021
Jan 9 2021
bin/update contains
reword commit title
dachary retitled D4832: bin/update fails unless pre-commit is installed from bin/update fails unless pip install pre-commit to bin/update fails unless pre-commit is installed.
dachary retitled D4832: bin/update fails unless pre-commit is installed from bin/update unless pip install pre-commit to bin/update fails unless pip install pre-commit.
Jan 8 2021
Jan 8 2021
forgot the leading (swh)