In T2309#87779, @douardda wrote:

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

@vsellier already did this to benchmark cassandra. it's indeed necessary to see how the backends behave with real loader and vault workloads. (less so for the objstorage, since the workloads should be much more uniform)

I'mt not exactly sure why I thought that would be necessary for benchmarking. In any case... it's not ;-)

The documentation is at:

It's documented in the winery test environment and was actually able to use the instructions successfully (after a few fixes...). It does work an this can be closed as resolved.

Added a wiki page to be a more accessible version of the benchmark process than the README in the sources.

In D6834#177663, @vlorentz wrote:

Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.

Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.

I don't understand why. Currently, we have: client --> objstorage pathslicing backend (port 5003) --> disk.
What you want to do is: client --> objstorage proxy (port 5003) --> objstorage winery backend (port 5012) --> winery --> ceph, right?
(objstorage winery backend is what is launched by docker/services/swh-winery/entrypoint.sh in this diff)

In D6834#177606, @vlorentz wrote:

Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra

Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra

It depends on https://forge.softwareheritage.org/D6796 and will fail until it is merged. It can be tested from sources with an override like this:

In practical terms, the two winery objstorage database servers and Ceph itself will be hosted at CEA, while the main ingestion storage / graph storage / ... will remain in Rocquencourt (separated sites, with fairly high bandwidth networking between them).