Throttling writes to 120MBs to reduce the pressure:

The number of slow random reads reaches ~3.5% presumably because there is too much write pressure (the throttling of writes was removed).

The benchmarks were modified to (i) use a fixed number of random / sequential readers instead of a random choice for better predictability, (ii) introduce throttling to cap the sequential reads speed to approximately 200MB/s. A run of read only was run:

The run terminated August 11th @ 15:21 because of what appears to be a rare race condition. It was however mostly finished. The results show an unexpected degradation in the read performances. It deserves further investigation because it keeps degrading over time. The write performance are however stable and suggest the benchmark code itself may be responsible for this degradation. If the Ceph cluster was globally slowing down, both reads and writes would show a degradation in performance because previous benchmark results showed that there is a correlation between the two.

Improve the readability of the graphs

Rehearse the run and make minor updates to make sure it runs right away this friday.

In the global read index, I would consider storing, for each object, alongside the shard id, the length and offset of the object (which are comparatively cheap to store)

The Compress, Hash and Displace: CHD Algorithm described in http://cmph.sourceforge.net/papers/wads07.pdf generates a hash function under 4MB for ~30M keys, 32 bytes each.

A 100GB file can have 25M objects (4KB median size). If a perfect hash function requires 4bits per entry, that's reading ~12MB for every lookup.

the colliding entries may be stored adjacent to each other...

I just realized that since a perfect hash function need parameters that may require additional sequential reads at the beginning of the file, it would actually make more sense to have a regular hash function with a format that allows for collisions. Even if the collisions are relatively frequent, the colliding entries may be stored adjacent to each other and will not require an additional read. They are likely to be in the same block most of the time. That would save the trouble of implementing a perfect hash function.

what are "the parameters to the perfect hash functions"? what are the possible formats?

The content of a file:

On the topic of throttling, the following discussion happened on IRC:

I misrepresented @olasd suggestions, here is the chat log on the matter.

In D6006#154829, @vlorentz wrote:

why *args, **kwargs on all methods?