Differential D7258 Diff 27232 sysadm/mirror-operations/docker.rst

Changeset View

Standalone View

View Options

sysadm/mirror-operations/docker.rst

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines

If you want to deploy services like the ``objstorage`` on several hosts, you will need a

shared storage area in which blob objects will be stored. Typically a NFS storage can be

used for this, or any existing docker volume driver like `REX-Ray

<https://rexray.readthedocs.io/>`_. This is not covered in this documentation.

Please read the documentation of docker volumes to learn how to use such a

device/driver as volume provider for docker.

Note that the provided :file:`base-services.yaml` file has placement constraints for the

``db-storage``, ``db-web`` and ``objstorage`` containers, that depend on the availability of

Node labels

specific volumes (respectively ``<STACK>_storage-db``, ``<STACK>_web-db`` and

-----------

``<STACK>_objstorage``). These services are pinned to specific nodes using labels named

Note that the provided :file:`base-services.yaml` file has label-based

placement constraints for several services.

ardumontUnsubmitted

Done

placement constraints for several services.

- The ``db-storage``, ``db-web``, ``objstorage`` and ``redis`` containers, that

+ The ``db-storage``, ``db-web``, ``objstorage`` and ``redis`` containers, which

depend on the availability of specific volumes (respectively

ardumont:

The ``db-storage``, ``db-web``, ``objstorage`` and ``redis`` containers, which

depend on the availability of specific volumes (respectively

``<STACK>_storage-db``, ``<STACK>_web-db`` and ``<STACK>_objstorage``) are

pinned to specific nodes using labels named

``org.softwareheritage.mirror.volumes.<base volume name>`` (e.g.

``org.softwareheritage.mirror.volumes.objstorage``).

When you create a local volume for a given container, you should add the relevant label

to the docker swarm node metadata with:

.. code-block:: bash

docker node update \

--label-add org.softwareheritage.mirror.volumes.objstorage=true \

<node_name>

You have to set the node labels, or to adapt the placement constraints to your local

requirements, for the services to start.

The monitoring services, ``prometheus``, ``prometheus-statsd-exporter`` and

``grafana`` also have placement constraints based on the label

``org.softwareheritage.mirror.monitoring``. So make sure to add this label to

one (and only one) node of the cluster:

.. code-block:: bash

docker node update \

--label-add org.softwareheritage.mirror.monitoring=true \

<node_name>

To check labels defined on a specific node, one can use the ``docker node

inspect`` command:

.. code-block:: bash

docker node inspect \

-f '{{ .ID }} [{{ .Description.Hostname}}]: \

{{ range $k, $v := .Spec.Labels }}{{ $k }}={{ $v }}

{{end}}' <node_name>

Labels that need to be defined are:

- ``org.softwareheritage.mirror.volumes.objstorage=true``: node that will host

the objstorage service, on which the ``swh_objstorage`` volume must be

defined.

- ``org.softwareheritage.mirror.volumes.redis=true``: node that will host the

redis service on which the ``swh_redis`` volume must be defined.

- ``org.softwareheritage.mirror.volumes.storage-db=true``: node that will host

the swh-storage Postgresql database, on which the ``swh_storage-db`` volume must

be defined.

- ``org.softwareheritage.mirror.volumes.web-db=true``: node that will host the

swh-web Postgresql database, on which the ``swh_web-db`` must be defined.

Managing secrets

----------------

Shared passwords (between services) are managed via :command:`docker secret`. Before

being able to start services, you need to define these secrets.

Namely, you need to create a ``secret`` for:

Show All 30 Lines

To specify the tag to be used, simply set the :envvar:`SWH_IMAGE_TAG`

environment variable:

.. code-block:: bash

~/swh-docker$ export SWH_IMAGE_TAG=20211022-121751

You can then spawn the base services using the following command:

Make sure you have node labels attributed properly. Then you can spawn the

base services using the following command:

ardumontUnsubmitted

Done

~/swh-docker$ export SWH_IMAGE_TAG=20211022-121751

- Make sure you have node labels attributed properly. Then you can then spawn the

+ Make sure you have node labels attributed properly. Then you can spawn the

base services using the following command:

ardumont:

.. code-block:: bash

~/swh-docker$ docker stack deploy -c base-services.yml swh

~/swh-docker$ docker stack deploy -c mirror.yml swh

Creating network swh_default

Creating config swh_content-replayer

Creating config swh_grafana-provisioning-datasources-prometheus

Creating config swh_graph-replayer

Creating config swh_grafana-provisioning-dashboards-all

Creating config swh_grafana-dashboards-content-replayer

Creating config swh_grafana-dashboards-backend-stats

Creating config swh_prometheus

Creating config swh_prometheus-statsd-exporter

Creating config swh_storage

Creating config swh_objstorage

Creating config swh_nginx

Creating config swh_web

Creating service swh_grafana

Creating config swh_grafana-dashboards-graph-replayer

Creating service swh_prometheus-statsd-exporter

Creating config swh_objstorage

Creating service swh_storage

Creating service swh_redis

Creating service swh_content-replayer

Creating service swh_nginx

Creating service swh_prometheus

Creating service swh_web

Creating service swh_prometheus-statsd-exporter

Creating service swh_db-web

Creating service swh_objstorage

Creating service swh_db-storage

Creating service swh_graph-replayer

Creating service swh_memcache

Creating service swh_storage

Creating service swh_grafana

Creating service swh_nginx

Creating service swh_prometheus

~/swh-docker$ docker service ls

ID NAME MODE REPLICAS IMAGE PORTS

tc93talbe2tg swh_db-storage global 1/1 postgres:13

ptlhzue025zm swh_content-replayer replicated 0/0 softwareheritage/replayer:20220225-101454

42q5jtxsh029 swh_db-web global 1/1 postgres:13

ycyanvhh0jnt swh_db-storage replicated 1/1 (max 1 per node) postgres:13

rtlz62ok6s96 swh_grafana replicated 1/1 grafana/grafana:latest

qlaf9tcyimz7 swh_db-web replicated 1/1 (max 1 per node) postgres:13

jao3rt0et17n swh_memcache replicated 1/1 memcached:latest

aouw9j8uovr2 swh_grafana replicated 1/1 (max 1 per node) grafana/grafana:latest

rulxakqgu2ko swh_nginx replicated 1/1 nginx:latest *:5081->5081/tcp

uwqe13udgyqt swh_graph-replayer replicated 0/0 softwareheritage/replayer:20220225-101454

q560pvw3q3ls swh_objstorage replicated 2/2 softwareheritage/base:20211022-121751

mepbxllcxctu swh_memcache replicated 1/1 memcached:latest

a2h3ltaqdt56 swh_prometheus global 1/1 prom/prometheus:latest

kfzirv0h298h swh_nginx global 3/3 nginx:latest *:5081->5081/tcp

lm24et9gjn2k swh_prometheus-statsd-exporter replicated 1/1 prom/statsd-exporter:latest

t7med8frg9pr swh_objstorage replicated 2/2 softwareheritage/base:20220225-101454

gwqinrao5win swh_storage replicated 2/2 softwareheritage/base:20211022-121751

5s34wzo29ukl swh_prometheus replicated 1/1 (max 1 per node) prom/prometheus:latest

7g46blmphfb4 swh_web replicated 1/1 softwareheritage/web:20211022-121751

rwom7r3yv5ql swh_prometheus-statsd-exporter replicated 1/1 (max 1 per node) prom/statsd-exporter:latest

wuwydthechea swh_redis replicated 1/1 (max 1 per node) redis:6.2.6

jztolbmjp1vi swh_storage replicated 2/2 softwareheritage/base:20220225-101454

xxc4c66x0uj1 swh_web replicated 1/1 softwareheritage/web:20220225-101454

This will start a series of containers with:

- an objstorage service,

- a storage service using a postgresql database as backend,

- a web app front end using a postgresql database as backend,

- a memcache for the web app,

- a prometheus monitoring app,

- a prometeus-statsd exporter,

- a grafana server,

- an nginx server serving as reverse proxy for grafana and swh-web.

- a swh_content-replayer service (initially set to 0 replica, see below)

- a swh_graph-replayer service (initially set to 0 replica, see below)

ardumontUnsubmitted

Done

- an nginx server serving as reverse proxy for grafana and swh-web.

- - a swh_content-replayer service (initially set to 0 replicas, see below)

- - a swh_graph-replayer service (initially set to 0 replicas, see below)

+ - a swh_content-replayer service (initially set to 0 replica, see below)

+ - a swh_graph-replayer service (initially set to 0 replica, see below)

- a redis for the replication error logs,

using the pinned version of the docker images.

ardumont:

- a redis for the replication error logs,

using the pinned version of the docker images.

The nginx frontend will listen on the 5081 port, so you can use:

- http://localhost:5081/ to navigate your local copy of the archive,

- http://localhost:5081/grafana/ to explore the monitoring probes

(log in with admin/admin).

▲ Show 20 Lines • Show All 77 Lines • ▼ Show 20 Lines

.. code-block:: bash

~/swh-docker$ export SWH_IMAGE_TAG=<new version>

~/swh-docker$ docker stack deploy -c base-services.yml swh

Note that this will reset the replicas config to their default values.

If you want to update only a specific service, you can also use (here for a

replayer service):

.. code-block:: bash

~/swh-docker$ docker service update --image \

softwareheritage/replayer:${SWH_IMAGE_TAG} \

swh_graph-replayer

.. warning::

Updating the image of a storage service may come with a database migration

script. So we strongly recommend you scale the service back to one before

updating the image:

.. code-block:: bash

~/swh-docker$ docker service scale swh_storage=1

~/swh-docker$ docker service update --image \

softwareheritage/base:${SWH_IMAGE_TAG} \

swh_storage

~/swh-docker$ docker service scale swh_storage=16

Set up the mirroring components

===============================

A Software Heritage mirror consists in base Software Heritage services, as

described above, without any worker related to web scraping nor source code

repository loading. Instead, filling local storage and objstorage is the

responsibility of kafka based ``replayer`` services:

- the ``graph replayer`` which is in charge of filling the storage (aka the

graph), and

- the ``content replayer`` which is in charge of filling the object storage.

Examples of docker deploy files and configuration files are provided in

The examples docker deploy file ``mirror.yml`` already define these 2

the :file:`graph-replayer.yml` deploy file for replayer services

services, but they are not deployed by default (their ``replicas`` is set to

using configuration from yaml files in :file:`conf/graph-replayer.yml`.

0). This allows to first deploy core components and check they are properly

started and running.

Copy these example files as plain yaml ones then modify them to replace

ardumontUnsubmitted

Done

started and running.

- To start the replayers, first their configuration files needs to be adjusted to

+ To start the replayers, first their configuration files need to be adjusted to

your setup.

ardumont:

the XXX markers with proper values (also make sure the kafka server list

To start the replayers, first their configuration files need to be adjusted to

is up to date). The parameters to check/update are:

your setup.

ardumontUnsubmitted

Done

- the ``content replayer`` which is in charge of filling the object storage.

- The examples docker deploy file ``mirror.yml`` already defines these 2

+ The examples docker deploy file ``mirror.yml`` already define these 2

services, but they are not deployed by default (their ``replicas`` is set to

0). This allows to first deploy core components and check they are properly

ardumont:

Edit the provided example files ``conf/graph-replayer.yml`` and

``conf/content-replayer.yml`` to modify fields with an XXX markers with proper

values (also make sure the kafka server list is up to date). The parameters to

check/update are:

- ``journal_client.brokers``: list of kafka brokers.

- ``journal_client.group_id``: unique identifier for this mirroring session;

you can choose whatever you want, but changing this value will make kafka

start consuming messages from the beginning; kafka messages are dispatched

among consumers with the same ``group_id``, so in order to distribute the

load among workers, they must share the same ``group_id``.

- ``journal_client.sasl.username``: kafka authentication username.

- ``journal_client.sasl.password``: kafka authentication password.

Then you need to merge the compose files "by hand" (due to this still

Then you need to update the configuration, as described above:

`unresolved <https://github.com/docker/cli/issues/1651>`_

`bugs <https://github.com/docker/cli/issues/1582>`_). For this we will use

`docker compose <https://github.com/docker/compose>`_ as helper tool to merge the

compose files.

To merge 2 (or more) compose files together, typically :file:`base-services.yml` with

a mirror-related file:

.. code-block:: bash

~/swh-docker$ docker-compose \

~/swh-docker$ docker config create swh_graph-replayer-2 conf/graph-replayer.yml

-f base-services.yml \

~/swh-docker$ docker service update \

-f graph-replayer-override.yml \

--config-rm swh_graph-replayer \

config > mirror.yml

--config-add source=swh_graph-replayer-2,target=/etc/softwareheritage/config.yml \

swh_graph-replayer

Then use this generated file as argument of the :command:`docker stack deploy`

and

command, e.g.:

.. code-block:: bash

~/swh-docker$ docker stack deploy -c mirror.yml swh

~/swh-docker$ docker config create swh_content-replayer-2 conf/content-replayer.yml

~/swh-docker$ docker service update \

--config-rm swh_content-replayer \

--config-add source=swh_content-replayer-2,target=/etc/softwareheritage/config.yml \

swh_content-replayer

Graph replayer

--------------

To run the graph replayer component of a mirror:

To run the graph replayer component of a mirror is just a matter of scaling its service:

.. code-block:: bash

~/swh-docker$ cd conf

~/swh-docker$ docker service scale swh_graph-replayer=1

~/swh-docker/conf$ cp graph-replayer.yml.example graph-replayer.yml

~/swh-docker/conf$ $EDITOR graph-replayer.yml

~/swh-docker/conf$ cd ..

Once you have properly edited the :file:`conf/graph-replayer.yml` config file,

you can start these services with:

.. code-block:: bash

~/swh-docker$ docker-compose \

-f base-services.yml \

-f graph-replayer-override.yml \

config > stack-with-graph-replayer.yml

~/swh-docker$ docker stack deploy \

-c stack-with-graph-replayer.yml \

swh

[...]

You can check everything is running with:

.. code-block:: bash

~/swh-docker$ docker stack ls

~/swh-docker$ docker service ps swh_graph-replayer

NAME SERVICES ORCHESTRATOR

swh 11 Swarm

~/swh-docker$ docker service ls

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS

ioyt34ok118a swh_graph-replayer.1 softwareheritage/replayer:20220225-101454 node1 Running Running 17 minutes ago

ID NAME MODE REPLICAS IMAGE PORTS

tc93talbe2tg swh_db-storage global 1/1 postgres:13

42q5jtxsh029 swh_db-web global 1/1 postgres:13

rtlz62ok6s96 swh_grafana replicated 1/1 grafana/grafana:latest

7hvn66um77wr swh_graph-replayer replicated 4/4 softwareheritage/replayer:20211022-121751

jao3rt0et17n swh_memcache replicated 1/1 memcached:latest

rulxakqgu2ko swh_nginx replicated 1/1 nginx:latest *:5081->5081/tcp

q560pvw3q3ls swh_objstorage replicated 2/2 softwareheritage/base:20211022-121751

a2h3ltaqdt56 swh_prometheus global 1/1 prom/prometheus:latest

lm24et9gjn2k swh_prometheus-statsd-exporter replicated 1/1 prom/statsd-exporter:latest

gwqinrao5win swh_storage replicated 2/2 softwareheritage/base:20211022-121751

7g46blmphfb4 swh_web replicated 1/1 softwareheritage/web:20211022-121751

If everything is OK, you should have your mirror filling. Check docker logs:

.. code-block:: bash

~/swh-docker$ docker service logs swh_graph-replayer

[...]

or:

.. code-block:: bash

~/swh-docker$ docker service logs --tail 100 --follow swh_graph-replayer

[...]

Content replayer

----------------

Similarly, to run the content replayer:

.. code-block:: bash

~/swh-docker$ cd conf

~/swh-docker$ docker service scale swh_content-replayer=1

~/swh-docker/conf$ cp content-replayer.yml.example content-replayer.yml

~/swh-docker/conf$ # edit content-replayer.yml files

~/swh-docker/conf$ cd ..

Once you have properly edited the :file:`conf/content-replayer.yml` config file, you can

start these services with:

.. code-block:: bash

~/swh-docker$ docker-compose \

-f base-services.yml \

-f content-replayer-override.yml \

config > content-replayer.yml

~/swh-docker$ docker stack deploy \

-c content-replayer.yml \

swh

[...]

Full mirror

-----------

Putting all together is just a matter of merging the 3 compose files:

.. code-block:: bash

~/swh-docker$ docker-compose \

-f base-services.yml \

-f graph-replayer-override.yml \

-f content-replayer-override.yml \

config > mirror.yml

~/swh-docker$ docker stack deploy \

-c mirror.yml \

swh

[...]

Getting your deployment production-ready

========================================

docker-stack scaling

--------------------

In order to scale up a replayer service, you can use the :command:`docker

Once the replayer services have been checked, started and are working

scale` command. For example:

properly, you can increase the replication to speed up the replication process.

ardumontUnsubmitted

Done

--------------------

- Once the replayer services has been checked and are starting and working

+ Once the replayer services have been checked and are started and working

properly, you can increase the replication to speed up the replication process.

ardumont:

.. code-block:: bash

~/swh-docker$ docker service scale swh_graph-replayer=4

~/swh-docker$ docker service scale swh_graph-replayer=64

[...]

~/swh-docker$ docker service scale swh_content-replayer=64

A proper replication factor value will depend on your infrastructure

capabilities and needs to be adjusted watching the load of the core services

(mainly the swh_storage-db and swh_objstorage services).

will start 4 copies of the graph replayer service.

Acceptable range should be between 32 to 64 (for staging) or 256 (for production).

Note that when you increase the replication of the replayers, you also need to

increase the replication factor for the core services ``swh_storage`` and

ardumontUnsubmitted

Done

increase the replication factor for the core services ``swh_storage`` and

- ``swh_objstorage`` otherwise they be the limitation factor of the replaying

+ ``swh_objstorage`` otherwise they will become the limitation factor of the replaying

process. A factor of 4 between the number of replayer of a type (graph,

ardumont:

``swh_objstorage`` otherwise they will become the limiting factor of the replaying

process. A factor of 4 between the number of replayer of a type (graph,

content) and the backend service (swh_storage, swh_objstorage) is probably a

good starting point (i.e. have at least one core service for 4 replayer

ardumontUnsubmitted

Done

good starting point (i.e. have at least one core service for 4 replayer

- service). You may have to play a bit with these values to find the sweet point.

+ services). You may have to play a bit with these values to find the right balance.

Notes on the throughput of the mirroring process

ardumont:

services). You may have to play a bit with these values to find the right balance.

Notes on the throughput of the mirroring process

------------------------------------------------

- One graph replayer service requires a steady 500MB to 1GB of RAM to run, so

make sure you have properly sized machines for running these replayer

containers, and to monitor these.

Show All 16 Lines

have it managed within the docker stack. To do so, you will have to:

- modify the (merged) configuration of the docker stack to drop references to the

``db-storage`` service (itself, and as dependency for the ``storage`` service)

- ensure that docker containers deployed in your swarm are able to connect to your

external database server

- override the environment variables of the ``storage`` service to reference the external

database server and dbname

Operational concerns for the monitoring

---------------------------------------

You may want to use a prometheus server running directly on one of the docker

swarm nodes so that it can easily also monitor the swarm cluster itself and the

running docker services.

See the `prometheus guide <https://prometheus.io/docs/guides/dockerswarm>`_ on

ardumontUnsubmitted

Done

See the `prometheus guide <https://prometheus.io/docs/guides/dockerswarm>`_ on

- how to configure a prometheus server to monitor a doscker swarm cluster.

+ how to configure a prometheus server to monitor a docker swarm cluster.

In this case, the ``prometheus`` service should be removed from the docker

ardumont:

how to configure a Prometheus server to monitor a docker swarm cluster.

In this case, the ``prometheus`` service should be removed from the docker

deploy compose file, and the configuration files should be updated accordingly.

You would probably want to move ``grafana`` from the docker swarm, and rework

the ``prometheus-statsd-exporter`` node setup accordingly.