Page MenuHomeSoftware Heritage

docker/test: add a pytest based test for the git loading stack
ClosedPublic

Authored by douardda on Jan 17 2020, 4:12 PM.

Details

Summary

For this, we extract the docker-compose setup in a dedicated fixture.
This is needed otherwise one cannot run all tests at once (both scheduler_host
and deposit_host conflict on the docker-compose part).

Also no need for most of the fixtures to be session scoped.

In docker/scheduler/entrypoint.sh, run setup_pgsql before setup_pip;
this hack is needed to prevent some race conditions in tests where it
sometimes happens that the ~/.pg_service file generated by the setup_pgsql
bash function is not visible in first calls to scheduler_host.check_output()
testinfra helper function.

Running setup_pip after setup_pgsql ensures(?) this does not happen.

Diff Detail

Repository
rDENV Development environment
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jan 17 2020, 4:12 PM

Why does setup_pip call scheduler_host.check_output()?

ardumont added inline comments.
docker/tests/conftest.py
41

stop

62

According to the description and the current code, this comment should go away.

Why does setup_pip call scheduler_host.check_output()?

it does not. My commit message is probably not very clear, but I've had cases where the execution of test_git_loader() fails with an error due to the fact when the git_origin fixture is executed, the .pg_services file does not yet exists (remember this is executed in a dedicated container started with 'docker-compose run' from within scheduler_host() fixture), so the 'swh scheduler task add' command fails because it cannot grate the scheduler object.

This hack (execute setup_pip after setup_pgsql) seems to do the trick and I do not want to try to understand why. So meh.

ardumont accepted this revision.Jan 17 2020, 5:36 PM

lgtm

docker/tests/test_git_loader.py
54

consistent

This revision is now accepted and ready to land.Jan 17 2020, 5:36 PM