How practical would it really be to use xz-file-format to create a 1TB archive with artifacts that have 3KB median size and 80KB average size ?
The index does not fit in memory and needs sorting.
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T3116 Roll out at least one operational mirror | ||
| Migrated | gitlab-migration | T3054 Scale out object storage design | ||
| Migrated | gitlab-migration | T3048 Using a custom Sorted String Table format | ||
| Migrated | gitlab-migration | T3046 Using xz-file-format for 1TB archive |
https://py7zr.readthedocs.io/en/latest/archive_format.html
The 7z format is more complex because it knows about files, directories etc. It is not not just a compressed data format.
https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md
The zstd format is tightly associated with the compression algorithm and is therefore more complex. It can however be a sequence of independently compressed content and could be used for the same purpose as xz.
https://www.nongnu.org/lzip/xz_inadequate.html
Xz format inadequate for long-term archiving
Although simple and close to what is needed, Xz is not an exact match: the index would need to be maintained.