I'd like to create a new package ( swh-objstorage-hash) and https://docs.softwareheritage.org/devel/tutorials/add-new-package.html is presumably the guide to do that. I however do not have the required permissions: would someone be so kind as to work with me on this?

Trvial bugfix https://forge.softwareheritage.org/D6417

it make sense to create a dedicated swh-perfecthash package.

That I did not know, so indeed, if we need a specific wrapper for our needs, ...

In addition to being unmaintained,

this could be addressed by asking authors to be in charge of the package

In T3104#71609, @dachary wrote:

@douardda

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

If I understand correctly, you're suggesting that I create a package at the same level as https://forge.softwareheritage.org/source/puppet-swh-site/, right ? For instance https://forge.softwareheritage.org/source/swh-perfecthash/ by following the instructions from the documentation.

So does it make sense to use this package instead of reimplementing one? What's the catch?

In addition to being unmaintained,

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

In T3104#71460, @dachary wrote:

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

It would but who would maintain it in the long run ?

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

In T3104#71408, @douardda wrote:

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

Or use this one maybe https://github.com/GregBowyer/cmph-cffi ?

Source for the cmph-cffi package in pypi seems to be https://github.com/venkateshks/cmph-cffi (well at least there are tags in there)

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

@olasd @douardda @thomash05 : the following passes tox -e py3 therefore it is not complete nonsense. However it raises two questions:

Throttling writes to 120MBs to reduce the pressure:

The number of slow random reads reaches ~3.5% presumably because there is too much write pressure (the throttling of writes was removed).

The benchmarks were modified to (i) use a fixed number of random / sequential readers instead of a random choice for better predictability, (ii) introduce throttling to cap the sequential reads speed to approximately 200MB/s. A run of read only was run:

The run terminated August 11th @ 15:21 because of what appears to be a rare race condition. It was however mostly finished. The results show an unexpected degradation in the read performances. It deserves further investigation because it keeps degrading over time. The write performance are however stable and suggest the benchmark code itself may be responsible for this degradation. If the Ceph cluster was globally slowing down, both reads and writes would show a degradation in performance because previous benchmark results showed that there is a correlation between the two.

well this task should be closed, and a new subtask could be added for the alerting

We should probably add monitoring alerts (if we don't already have them) before closing the task

unless I'm mistaken, this task can be closed now, it looks to have reached a steady state where the lag is near 0

Improve the readability of the graphs

Rehearse the run and make minor updates to make sure it runs right away this friday.

In the global read index, I would consider storing, for each object, alongside the shard id, the length and offset of the object (which are comparatively cheap to store)