I have found some interesting pointers relative to the management of small files in hdfs (found them when looking for unrelated other stuff). Is it something you have identified and excluded from the scope due to some blockers ?

• medium size files management (100k -> 10Gb): https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf
• small files using readonly archive, which looks very similar to your proposal, aka having a live/cold storage areas):
  • https://docs.cloudera.com/runtime/7.2.2/scaling-namespaces/topics/hdfs-optimizing-namenode-disk-space-with-hadoop-archives.html
  • https://blog.cloudera.com/the-small-files-problem/
  • https://pastiaro.wordpress.com/2013/06/05/solving-the-small-files-problem-in-apache-hadoop-appending-and-merging-in-hdfs/
  • ...

Very interesting to see how this problem was presented & solved in the Hadoop ecosystem, thanks for the links.

• https://github.com/apache/hbase/blob/master/dev-support/design-docs/HBASE-11339%20MOB%20GA%20design.pdf does not explain the problem it is trying to solve which makes it difficult to relate to the Software Heritage use case. It is very HDFS/HBASE specific in how it is implemented.

• HAR (Hadoop Archives) were designed to reduce the HDFS space amplification for small objects (no specifics on how much amplification that is exactly). This 2009 article gives a hint: every HDFS file requires 150 bytes in RAM and it states that "Certainly a billion files is not feasible." and that explains why HAR was introduced.

Thanks for helping with the labelling @rdicosmo 👍

Bookmarking https://leo-project.net/leofs/

https://git.easter-eggs.org/biceps/biceps/-/tree/c27d80e00e206413a1b07b70806cb38cb702fe48

Improve the readability of the graphs