A possible optimization could be to generate a https://en.wikipedia.org/wiki/Perfect_hash_function
The custom format is a header:
- Format version
followed by an index which is a hash table
after the index the content of the objects is found.
It is assumed writing is done in batch, sequentially
- HASH(SHA256) in the index
- Seek to the object content to stream it to the caller in chunks of a given size