Add an optional swh.storage read replica to the docker setup
ClosedPublic
Actions

Authored by olasd on Sep 24 2020, 12:53 PM.

Details

Reviewers

ardumont
douardda

Commits

rDENVf64cd5eff72b: Add an optional swh.storage read replica to the docker setup
rDENVca8f6ab266b1: Support overriding the host in wait_pgsql
rDENV28c10e591e60: Make sure to explicitly empty .pg_service.conf before adding entries

Summary

Summary
This configuration adds a new swh.storage database and backend using logical
postgresql replication. On startup of the read-replica storage backend, the
replication is configured between both database hosts.

The configuration also overrides the swh.web and swh.vault containers to use the
new read-only replica.

Two extra scaffolding commits:

Make sure to explicitly empty .pg_service.conf before adding entries
Support overriding the host in wait_pgsql

Depends on D4022
Depends on D4024

(also requires the changes to swh.storage SQL in D3981)

Test Plan

docker-compose -f docker-compose.yml -f docker-compose.storage-read-replica.yml -f docker-compose.override.yml up -d
export SWH_SCHEDULER_URL=http://127.0.0.1:5008/
swh scheduler task add load-git -p oneshot url=https://github.com/Microsoft/MS-DOS

And then clickity click on the web interface to try to cook one of the directories.

Diff Detail

Repository

rDENV Development environment

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Sep 24 2020, 12:53 PM

Harbormaster completed remote builds in B15494: Diff 14218.Sep 24 2020, 12:53 PM

ardumont added inline comments.Sep 24 2020, 2:27 PM

docker/docker-compose.storage-read-replica.yml
62 ↗	(On Diff #14218)	i don't see a cooker-read-replica.yml only vault-worker-read-replica.yml which seemed relevant here.
docker/services/swh-storage-read-replica/entrypoint.sh
21 ↗	(On Diff #14218)	given that the docker-compose*.yml introduced does not expose such backend, it must be preferable to remove it from the entrypoint... That'd be less confusing.

lgtm, one point of attention inline (about the vault worker configuration file).

docker/services/swh-storage-read-replica/entrypoint.sh
38 ↗	(On Diff #14218)	\o/ That's nice ;)

This revision is now accepted and ready to land.Sep 24 2020, 2:40 PM

douardda accepted this revision.Sep 24 2020, 3:03 PM

olasd added inline comments.Sep 24 2020, 4:09 PM

docker/docker-compose.storage-read-replica.yml
62 ↗	(On Diff #14218)	Sure. See line below : vault-worker-read-replica.yml is mounted on cooker-read-replica.yml. (I only carried over the inconsistency from the other definition of the service...)
docker/services/swh-storage-read-replica/entrypoint.sh
21 ↗	(On Diff #14218)	I considered it, yeah.