Aug 23 2022
Jul 19 2022
Jul 1 2022
do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?
Jun 27 2022
Jun 21 2022
May 1 2022
Apr 29 2022
Apr 8 2022
Apr 5 2022
Mar 30 2022
Mar 25 2022
Mar 23 2022
Jan 25 2022
@vsellier already did this to benchmark cassandra. it's indeed necessary to see how the backends behave with real loader and vault workloads. (less so for the objstorage, since the workloads should be much more uniform)
I'mt not exactly sure why I thought that would be necessary for benchmarking. In any case... it's not ;-)
The documentation is at:
It's documented in the winery test environment and was actually able to use the instructions successfully (after a few fixes...). It does work an this can be closed as resolved.
Added a wiki page to be a more accessible version of the benchmark process than the README in the sources.
Jan 22 2022
Dec 16 2021
Dec 14 2021
Ah wait I got it, docker/services/swh-winery/entrypoint.sh launches winery itself, not an actual objstorage backend; you're only reusing the scafholding.
I don't understand why. Currently, we have: client --> objstorage pathslicing backend (port 5003) --> disk.
What you want to do is: client --> objstorage proxy (port 5003) --> objstorage winery backend (port 5012) --> winery --> ceph, right?
(objstorage winery backend is what is launched by docker/services/swh-winery/entrypoint.sh in this diff)
Instead of defining a new service, could you provide an alternative docker-compose config file? This way, it can be used as to switch all services to use it, just by adding a CLI parameter. eg. we do this to replace the postgres storage backend with cassandra: https://docs.softwareheritage.org/devel/getting-started/using-docker.html#cassandra
Dec 13 2021
In practical terms, the two winery objstorage database servers and Ceph itself will be hosted at CEA, while the main ingestion storage / graph storage / ... will remain in Rocquencourt (separated sites, with fairly high bandwidth networking between them).