https://en.wikipedia.org/wiki/Perfect_hash_function could be used to get O(1) instead of O(log(N))

Very interesting to see how this problem was presented & solved in the Hadoop ecosystem, thanks for the links.

I have found some interesting pointers relative to the management of small files in hdfs (found them when looking for unrelated other stuff). Is it something you have identified and excluded from the scope due to some blockers ?

For the record the half baked benchmark script for the proposed designed I worked on today. To be continued!

Reopening for benchmarking purposes because there does not seem to be anything ready to use T3068.

jumpDB is 100% python therefore less than ideal for CPU performance but for the purpose of benchmarking I/O and space usage it is conveniently ready to use.

There is not enough tooling to use SST files independently of RocksDB. Maybe it is possible to use the RocksDB with a configuration that makes it so it only uses a single SST file ?