Sun, May 1
Fri, Apr 29
Apr 8 2022
Apr 5 2022
Mar 30 2022
Mar 25 2022
Mar 23 2022
Jan 25 2022
@vsellier already did this to benchmark cassandra. it's indeed necessary to see how the backends behave with real loader and vault workloads. (less so for the objstorage, since the workloads should be much more uniform)
I'mt not exactly sure why I thought that would be necessary for benchmarking. In any case... it's not ;-)
The documentation is at:
It's documented in the winery test environment and was actually able to use the instructions successfully (after a few fixes...). It does work an this can be closed as resolved.
Added a wiki page to be a more accessible version of the benchmark process than the README in the sources.
Jan 22 2022
Dec 16 2021
Dec 14 2021
Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.
I don't understand why. Currently, we have: client --> objstorage pathslicing backend (port 5003) --> disk.
What you want to do is: client --> objstorage proxy (port 5003) --> objstorage winery backend (port 5012) --> winery --> ceph, right?
(objstorage winery backend is what is launched by docker/services/swh-winery/entrypoint.sh in this diff)
Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra
It depends on https://forge.softwareheritage.org/D6796 and will fail until it is merged. It can be tested from sources with an override like this:
Dec 13 2021
In practical terms, the two winery objstorage database servers and Ceph itself will be hosted at CEA, while the main ingestion storage / graph storage / ... will remain in Rocquencourt (separated sites, with fairly high bandwidth networking between them).
Dec 12 2021
@olasd I split the debian packaging in its own task at T3797 so that this task can be closed. I'll let you revert this if you think it is not appropriate. My rationale is that it would be easier to figure out what's left to be done with this one other task. Rather than coming back to this rather overloaded ticket. But it's just a matter of personal taste :-)