• move winery test environment at the root of the repository as suggested

In D6796#177502, @vlorentz wrote:

Hmm... alright, let's keep it in this repo. But move it to the root (ie. something like swh-objstorage/winery-benchmarks/ instead of swh-objstorage/swh/objstorage/tests/winery/), as the current location implies it is used from the Python code.

Sounds good?

In D6796#177485, @vlorentz wrote:

In D6796#177439, @dachary wrote:

Just so we're clear: these ansible files are uniquely used for setting up the test environment (local or fed4fire). Are you suggesting that part of the test environment is moved in the snippets repository? Or did you maybe think that these ansible files were not used for testing purposes?

Do you mean they are invoked via pytest?

• fix database closing ordering to not leak connections
• automatic connection cleanup on test failure

• finalize fed4fire scripts and obtained a successful test run
• address review comments
• fix missing drop of the Write Shard when packing completes

In D6796#177352, @vlorentz wrote:

Thanks!

Could you add comments in the code to address @vsellier's questions? (he probably won't be the last to ask them while reading the code)

resolve race condition that prevents updating /etc/hosts

nuke /dev/sdc early on so that lvm2 does not grab it

• rework the integration tests and documentation to be grid5000/fed4fire compatible

implement tox -e winery and associated helper script

move all files created for the ceph test cluster in a dedicated directory

@olasd I split the debian packaging in its own task at T3797 so that this task can be closed. I'll let you revert this if you think it is not appropriate. My rationale is that it would be easier to figure out what's left to be done with this one other task. Rather than coming back to this rather overloaded ticket. But it's just a matter of personal taste :-)

The documentation now shows as expected. The previous problems in rendering it were probably because the package was not published.

Duplicate of T3533

Duplicate of T3533

I think D6796 is ready for review and should be merged. It is not final and won't run in production but it is a working implementation of the object storage. From there I intend to setup a benchmarking environment on grid5000. Optimizations will be implemented to improve the benchmark results where it matters the most. The benchmarks that were run during the summer already provided insights which motivated the implementation of the shard packer in C.

add get from Read Shard and test

• assert the expected Readonly Shard packing state
• document the pytest-posgresql hang prevention tips

• ensure pytest-postgresql is able to cleanup on failure instead of hanging
• minimal test with swh.perfecthash

do not use pg13 specific statements

rework the fixtures to prevent hanging when a test fails

ceph --help actually exits with non zero, use --version instead

add test environment and tools for the Ceph cluster

add the packing boolean to the shards table

This is all news to me but it makes intuitive sense. Plus it was tested manually and I can testify it worked. So I review positively (yeah, I make a big deal out of this because this is my first review ;-) )

address review comments
remove ceph ssh test key from the repository, it is noise

helper script to spawn a container based Ceph cluster for integration testing in a CI environment

Thanks for the quick review :-)

sort

There is a need for wheels even for non arch dependent packages.

@ardumont @olasd I'm not sure if it was polite / useful to add you as subscribers to this isolated commit. Please let me know if I should have done otherwise.

It was tested as follows:

I thought it was pushed already, I must have overlooked it somehow. Doing it now.

I've read https://docs.softwareheritage.org/sysadm/deployment/howto-debian-packaging.html to figure out what should be done to complete the "Bootstrap Debian packaging" part which does not show in the pool https://debian.softwareheritage.org/pool/main/s/ . I think to understand this is done by a job in https://forge.softwareheritage.org/source/swh-jenkins-jobs/, presumably https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$196 but... it's not happening

D6796 is a draft implementation that is not ready for review but can be reviewed anyways in case someone is curious.