HomeSoftware Heritage

Control Docker more robustly

Description

Control Docker more robustly

Previously, Docker was controlled through custom shell commands issued
via pytest-testinfra. Multiple loops where used to wait for containers
to converge to the desired state. Given the test requires quite some
resources in terms of RAM, CPU, and bandwith, its execution can take
between 20 minutes and a full hour. Therefore, it would be better
to avoid having to edit the code to fiddle with timeouts and loop
counts.

One of the way we used to wait for completion was looking at the
Docker service logs for specific messages. Sadly, pytest-testinfra
does not really provide a way to monitor the output of a command as it
runs.

All-in-all, it felt easier to replace the custom shell commands by a
library offering a Python API to control Docker. While dockerpy is the
official library, its future maintainance by Docker has been in flux
(see: https://github.com/docker/docker-py/pull/2989) and more
importantly, as it speaks directly to the Docker socket, it does not
support anything like docker stack.

Meanwhile, the python_on_whales has been featured on the official
Docker blog (https://www.docker.com/blog/guest-post-calling-the-docker-cli-from-python-with-python-on-whales/)
and implements as much as possible of Docker command line interface (by
shelling out).

With it, it was possible to:

  • Replace all command lines with Python calls.
  • Wait for containers to shut down on teardown before removing volumes.
  • Wait for log messages to appear by blocking on the service output.
  • Wait for Docker readyness when we start the stack and scale services.

Some refactoring and naming improvements have been made along the way.
Some leftover breakpoints were also removed.

Because we still would like to timeout in case of a problem,
pytest-timeout has been added to the requirements with a default
timeout of 30 minutes.

Details

Provenance
lunarAuthored on Oct 4 2022, 4:33 PM
lunarPushed on Nov 7 2022, 1:38 PM
Differential Revision
D8634: Prepare the tests to run in Jenkins
Parents
rCDFPadc6295fe5dd: Restart graph replayer automatically
Branches
Unknown
Tags
Unknown