Page MenuHomeSoftware Heritage

docker: Fix journal client services
ClosedPublic

Authored by anlambert on Feb 16 2021, 7:27 PM.

Details

Summary

Services acting as journal clients were always exiting prematurely due
to not created kafka topics they were trying to subscribe to.

So ensure needed topics will be created when starting the kafka service
and wait for their creation before starting the journal client services.

As a consequence, this makes a call to docker-compose up -d command a
little bit longer before it returns.

Also fix a service configuration and enable DEBUG logging in another one
(see commit log).

Diff Detail

Repository
rDENV Development environment
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added a subscriber: ardumont.

awesome, thanks.

This revision is now accepted and ready to land.Feb 17 2021, 9:18 AM

Please note a recent version of docker-compose is needed to allow the environment to start with the healthcheck keywork,
It was not working with the version 1.26.2 but is ok with 1.28.2

❯ docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

❯ docker-compose -f docker-compose.yml -f docker-compose.search.yml -f docker-compose.override.yml up -d --force-recreate --no-dep swh-search
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.kafka: 'healthcheck'
services.swh-scheduler-journal-client.depends_on contains an invalid type, it should be an array
services.swh-indexer-journal-client.depends_on contains an invalid type, it should be an array
❯ docker-compose version
docker-compose version 1.28.2, build 67630359
docker-py version: 4.4.1
CPython version: 3.7.9
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019


❯ docker-compose -f docker-compose.yml -f docker-compose.search.yml -f docker-compose.override.yml up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Recreating docker_nginx_1                        ... done
...

(forgot to mention: Thanks, it's really a nice improvement)

Please note a recent version of docker-compose is needed to allow the environment to start with the healthcheck keywork,
It was not working with the version 1.26.2 but is ok with 1.28.2

❯ docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

❯ docker-compose -f docker-compose.yml -f docker-compose.search.yml -f docker-compose.override.yml up -d --force-recreate --no-dep swh-search
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.kafka: 'healthcheck'
services.swh-scheduler-journal-client.depends_on contains an invalid type, it should be an array
services.swh-indexer-journal-client.depends_on contains an invalid type, it should be an array
❯ docker-compose version
docker-compose version 1.28.2, build 67630359
docker-py version: 4.4.1
CPython version: 3.7.9
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019


❯ docker-compose -f docker-compose.yml -f docker-compose.search.yml -f docker-compose.override.yml up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Recreating docker_nginx_1                        ... done
...

Ah right, the feature was introduced in docker-compose 1.27.0 (see commit).

We need to find a way to restrict the version of docker-compose to use then, maybe by changing the version field value at top of docker-compose.yml file.

Ah right, the feature was introduced in docker-compose 1.27.0

Actually, healthcheck it is a feature introduced in version 2.1 of Compose file format.

Bumping version to 2.1 in compose files seems to fix the observed issue (just tested with docker-compose 1.25.0 packaged in debian buster).