diff --git a/README.md b/README.md index 439945d..856a1cd 100644 --- a/README.md +++ b/README.md @@ -1,302 +1,399 @@ # Deploy a Software Heritage stack with docker deploy +## Prerequisities + According you have a properly set up docker swarm cluster with support for the -`docker deploy` command, e.g.: +`docker stack deploy` command, e.g.: ``` ~/swh-docker$ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION py47518uzdb94y2sb5yjurj22 host2 Ready Active 18.09.7 n9mfw08gys0dmvg5j2bb4j2m7 * host1 Ready Active Leader 18.09.7 ``` -Note: this might require you activate experimental features of docker as -described in [docker deploy](https://docs.docker.com/engine/reference/commandline/deploy/) -documentation. +Note: on some systems (centos for example), making docker swarm works require +some permission tuning regarding the firewall or selinux. In the following how-to, we will assume that the service `STACK` name is `swh` -(this name is the last argument of the `docker deploy` command below). +(this name is the last argument of the `docker stack deploy` command below). Several preparation steps will depend on this name. +We also use [docker-compose](https://github.com/docker/compose) to merge +compose files, so make sure it iavailable on your system. + ## Set up volumes Before starting the `swh` service, you may want to specify where the data should be stored on your docker hosts. -By default it will use docker volumes for storing databases and the content of -the objstorage (thus put them in `/var/lib/docker/volumes`. +By default docker will use docker volumes for storing databases and the content of +the objstorage (thus put them in `/var/lib/docker/volumes`). -If you want to specify a different location to put a storage in, create the -storage before starting the docker service. For example for the `objstorage` -service you will need a storage named `_objstorage`: +**Optional:** if you want to specify a different location to put a storage in, +create the storage before starting the docker service. For example for the +`objstorage` service you will need a storage named `_objstorage`: ``` ~/swh-docker$ docker volume create -d local \ --opt type=none \ --opt o=bind \ --opt device=/data/docker/swh-objstorage \ swh_objstorage ``` If you want to deploy services like the `swh-objstorage` on several hosts, you -will a shared storage area in which blob objects will be stored. Typically a -NFS storage can be used for this. This is not covered in this doc. +will need a shared storage area in which blob objects will be stored. Typically +a NFS storage can be used for this, or any existing docker volume driver like +[REX_Rey](https://rexray.readthedocs.io/). This is not covered in this doc. Please read the documentation of docker volumes to learn how to use such a -device as volume proviver for docker. +device/driver as volume provider for docker. + +Note that the provided `base-services.yaml` file have a few placement +constraints: containers that depends on a volume (db-storage and objstorage) +are stick to the manager node of the cluster, under the assumption persistent +volumes have been created on this node. Make sure this fits your needs, or +change these placement constraints. -Note that the provided `docker-compose.yaml` file have a few placement -constraints, for example the `objstorage` service is forced to be spawn on the -master node of the docker swarm cluster. Feel free to remove/amend these -constraints if needed. ## Managing secrets Shared passwords (between services) are managed via `docker secret`. Before being able to start services, you need to define these secrets. Namely, you need to create a `secret` for: - `postgres-password` For example: ``` ~/swh-docker$ echo 'strong password' | docker secret create postgres-password - [...] ``` ## Creating the swh base services -From within this repository, just type: +If you haven't done it yet, clone this git repository: +``` +~$ git clone https://forge.softwareheritage.org/source/swh-docker.git +~$ cd swh-docker ``` -~/swh-docker$ docker deploy -c docker-compose.yml swh -Creating service swh_web -Creating service swh_objstorage -Creating service swh_storage -Creating service swh_nginx -Creating service swh_memcache -Creating service swh_db-storage + +then from within this repository, just type: + +``` +~/swh-docker$ docker stack deploy -c base-services.yml swh +Creating network swh-mirror_default +Creating config swh-mirror_storage +Creating config swh-mirror_objstorage +Creating config swh-mirror_nginx +Creating config swh-mirror_web +Creating service swh-mirror_grafana +Creating service swh-mirror_prometheus-statsd-exporter +Creating service swh-mirror_web +Creating service swh-mirror_objstorage +Creating service swh-mirror_db-storage +Creating service swh-mirror_memcache +Creating service swh-mirror_storage +Creating service swh-mirror_nginx +Creating service swh-mirror_prometheus ~/swh-docker$ docker service ls -ID NAME MODE REPLICAS IMAGE PORTS -bkn2bmnapx7w swh_db-storage replicated 1/1 postgres:11 -2ujcw3dg8f9d swh_memcache replicated 1/1 memcached:latest -l52hxxl61ijj swh_nginx replicated 1/1 nginx:latest *:5080->80/tcp -3okk2njpbopx swh_objstorage replicated 1/1 softwareheritage/base:latest -zais9ey62weu swh_storage replicated 1/1 softwareheritage/base:latest -7sm6g5ecff19 swh_web replicated 1/1 softwareheritage/web:latest +ID NAME MODE REPLICAS IMAGE PORTS +sz98tofpeb3j swh-mirror_db-storage global 1/1 postgres:11 +sp36lbgfd4qi swh-mirror_grafana replicated 1/1 grafana/grafana:latest +7oja81jngiwo swh-mirror_memcache replicated 1/1 memcached:latest +y5te0gqs93li swh-mirror_nginx replicated 1/1 nginx:latest *:5081->5081/tcp +79t3r3mv3qn6 swh-mirror_objstorage replicated 1/1 softwareheritage/base:20200918-133743 +l7q2zocoyvq6 swh-mirror_prometheus global 1/1 prom/prometheus:latest +p6hnd90qnr79 swh-mirror_prometheus-statsd-exporter replicated 1/1 prom/statsd-exporter:latest +jjry62tz3k76 swh-mirror_storage replicated 1/1 softwareheritage/base:20200918-133743 +jkkm7qm3awfh swh-mirror_web replicated 1/1 softwareheritage/web:20200918-133743 ``` This will start a series of containers with: - an objstorage service, - a storage service using a postgresql database as backend, - a web app front end, - a memcache for the web app, - a prometheus monitoring app, - a prometeus-statsd exporter, - a grafana server, - an nginx server serving as reverse proxy for grafana and swh-web. -using the latest published version of the docker images. +using the latest published version of the docker images by default. The nginx frontend will listen on the 5081 port, so you can use: - http://localhost:5081/ to navigate your local copy of the archive, - http://localhost:5081/grafana/ to explore the monitoring probes (log in with admin/admin). -Note that if the 'latest' docker images work, it is highly recommended to -explicitly specify the version of the image you want to use. +>[!WARNING] +>the 'latest' docker images work, it is highly recommended to +>explicitly specify the version of the image you want to use. + Docker images for the Software Heritage stack are tagged with their build date: docker images -f reference='softwareheritage/*:20*' REPOSITORY TAG IMAGE ID CREATED SIZE softwareheritage/web 20200819-112604 32ab8340e368 About an hour ago 339MB softwareheritage/base 20200819-112604 19fe3d7326c5 About an hour ago 242MB softwareheritage/web 20200630-115021 65b1869175ab 7 weeks ago 342MB softwareheritage/base 20200630-115021 3694e3fcf530 7 weeks ago 245MB To specify the tag to be used, simply set the SWH_IMAGE_TAG environment variable, like: export SWH_IMAGE_TAG=20200819-112604 - docker deploy -c docker-compose.yml swh + docker deploy -c base-services.yml swh -Warning: make sure to have this variable properly set for any later `docker deploy` -command you type, otherwise you running containers will be recreated using the -':latest' image (which might **not** be the latest available version, nor -consistent amond the docker nodes on you swarm cluster). +>[!WARNING] +>make sure to have this variable properly set for any later `docker deploy` +>command you type, otherwise you running containers will be recreated using the +>':latest' image (which might **not** be the latest available version, nor +>consistent amond the docker nodes on you swarm cluster). ## Updating a configuration When you modify a configuration file exposed to docker services via the `docker -config` system, you need to destroy the old config before being able to -recreate them (docker is currently not capable of updating an existing config.) -Unfortunately that also means you need to recreate every docker container using -this config. +config` system. Unfortunately, docker does not support updating these config +objects, so you need to destroy the old config before being able to recreate +them. That also means you need to recreate every docker container using this +config. For example, if you edit the file `conf/storage.yml`: ``` ~/swh-docker$ docker service rm swh_storage swh_storage ~/swh-docker$ docker config rm swh_storage swh_storage -~/swh-docker$ docker deploy -c docker-compose.yml swh +~/swh-docker$ docker stack deploy -c base-services.yml swh Creating config swh_storage Creating service swh_storage Updating service swh_nginx (id: l52hxxl61ijjxnj9wg6ddpaef) Updating service swh_memcache (id: 2ujcw3dg8f9dm4r6qmgy0sb1e) Updating service swh_db-storage (id: bkn2bmnapx7wgvwxepume71k1) Updating service swh_web (id: 7sm6g5ecff1979t0jd3dmsvwz) Updating service swh_objstorage (id: 3okk2njpbopxso3n3w44ydyf9) +[...] ``` See https://docs.docker.com/engine/swarm/configs/ for more details on -how to use the config system in docker swarm. +how to use the config system in a docker swarm cluster. Note that since persistent data (databases and objects) are stored in volumes, you can safely destoy and recreate any container you want, you will not loose any data. ## Updating a service When a new version of the softwareheritage/base image is published, running services must updated to use it. In order to prevent inconsistency caveats due to dependency in deployed versions, we recommend that you deploy the new image on all running services at once. This can be done as follow: ``` ~/swh-docker$ export SWH_IMAGE_TAG= -~/swh-docker$ docker deploy -c docker-compose.yml swh +~/swh-docker$ docker stack deploy -c base-services.yml swh ``` Note that this will reset the replicas config to their default values. If you want to update only a specific service, you can also use: ``` ~/swh-docker$ docker service update --image \ softwareheritage/base:${SWH_IMAGE_TAG} ) \ - swh_graph-replayer-origin + swh_graph-replayer ``` # Set up a mirror A Software Heritage mirror consists in base Software Heritage services, as described above, without any worker related to web scraping nor source code repository loading. Instead, filling local storage and objstorage is the responsibility of kafka based `replayer` services: - the `graph replayer` which is in charge of filling the storage (aka the graph), and - the `content replayer` which is in charge of filling the object storage. -Examples of docker-compose files and configuration files are provided in -the `graph-replayer-remote-bytopic.yml` compose file for replayer services -using configuration from yaml files in `conf/graph-replayer/remote/`. +Examples of docker deploy files and configuration files are provided in +the `graph-replayer.yml` deploy file for replayer services +using configuration from yaml files in `conf/graph-replayer.yml`. Copy these example files as plain yaml ones then modify them to replace the XXX merkers with proper values (also make sure the kafka server list is up to date.) Parameters to check/update are: - `journal_client/brokers`: list of kafka brokers. - `journal_client/group_id`: unique identifier for this mirroring session; you can choose whatever you want, but changing this value will make kafka start consuming messages from the beginning; kafka messages are dispatched among consumers with the same `group_id`, so in order to distribute the load among workers, they must share the same `group_id`. - `journal_client/sasl.username`: kafka authentication username. - `journal_client/sasl.password`: kafka authentication password. +Then you need to merge the compose files "by hand" (due to this still +[unresolved](https://github.com/docker/cli/issues/1651) +[bugs](https://github.com/docker/cli/issues/1582)). For this we will use +[docker-compose](https://github.com/docker/compose) as helper tool to merge the +compose files. + +To merge 2 (or more) compose files together, typically `base-services.yml` with +a mirror-related file: + +``` +~/swh-docker$ docker-compose \ + -f base-services.yml \ + -f graph-replayer-override.yml \ + config > merged-compose.yml +``` + +Then use this generated file as argument of the `docker stack deploy` command. + + +## Graph replayer + +To run the graph replayer: + ``` -~/swh-docker$ cd conf/graph-replayer/remote -~/swh-docker/conf/graph-replayer/remote$ for i in *.example; do cp $i ${i/.example//}; done -~/swh-docker/conf/graph-replayer/remote$ # edit .yml files -~/swh-docker/conf/graph-replayer/remote$ cd ../../.. -~/swh-docker$ +~/swh-docker$ cd conf +~/swh-docker/conf$ cp graph-replayer.yml.example graph-replayer.yml +~/swh-docker/conf$ # edit graph-replayer.yml files +~/swh-docker/conf$ cd .. ``` -Once you have properly edited config files, you can start these services with: +Once you have properly edited the `conf/graph-replayer.yml` config file, you can +start these services with: ``` -~/swh-docker$ docker deploy \ - -c docker-compose.yml \ - -c graph-replayer-remote-bytopic.yml \ +~/swh-docker$ docker-composer \ + -f base-services.yml \ + -f graph-replayer-override.yml \ + config > graph-replayer.yml +~/swh-docker$ docker stack deploy \ + -c graph-replayer.yml \ swh [...] ``` You can check everything is running with: ``` +~/swh-docker$ docker stack ls +NAME SERVICES ORCHESTRATOR +swh-mirror 11 Swarm ~/swh-docker$ docker service ls ID NAME MODE REPLICAS IMAGE PORTS 88djaq3jezjm swh_db-storage replicated 1/1 postgres:11 m66q36jb00xm swh_grafana replicated 1/1 grafana/grafana:latest qfsxngh4s2sv swh_content-replayer replicated 1/1 softwareheritage/base:latest -qcl0n3ngr2uv swh_graph-replayer-content replicated 2/2 softwareheritage/base:latest -f1hop14w6b9h swh_graph-replayer-directory replicated 4/4 softwareheritage/base:latest -dcpvbf7h4fja swh_graph-replayer-origin replicated 2/2 softwareheritage/base:latest -1njy5iuugmk2 swh_graph-replayer-release replicated 2/2 softwareheritage/base:latest -cbe600nl9bdb swh_graph-replayer-revision replicated 4/4 softwareheritage/base:latest -5hroiithan6c swh_graph-replayer-snapshot replicated 2/2 softwareheritage/base:latest +qcl0n3ngr2uv swh_graph-replayer replicated 2/2 softwareheritage/base:latest zn8dzsron3y7 swh_memcache replicated 1/1 memcached:latest wfbvf3yk6t41 swh_nginx replicated 1/1 nginx:latest *:5081->5081/tcp thtev7o0n6th swh_objstorage replicated 1/1 softwareheritage/base:latest ysgdoqshgd2k swh_prometheus replicated 1/1 prom/prometheus:latest u2mjjl91aebz swh_prometheus-statsd-exporter replicated 1/1 prom/statsd-exporter:latest xyf2xgt465ob swh_storage replicated 1/1 softwareheritage/base:latest su8eka2b5cbf swh_web replicated 1/1 softwareheritage/web:latest ``` If everything is OK, you should have your mirror filling. Check docker logs: ``` -~/swh-docker$ docker service logs swh_content-replayer +~/swh-docker$ docker service logs swh_graph-replayer [...] ``` or: ``` -~/swh-docker$ docker service logs --tail 100 --follow swh_graph-replayer-directory +~/swh-docker$ docker service logs --tail 100 --follow swh_graph-replayer +[...] +``` + + +## Graph replayer + +Similarly, to run the content replayer: + +``` +~/swh-docker$ cd conf +~/swh-docker/conf$ cp content-replayer.yml.example content-replayer.yml +~/swh-docker/conf$ # edit content-replayer.yml files +~/swh-docker/conf$ cd .. + +``` + +Once you have properly edited the `conf/content-replayer.yml` config file, you can +start these services with: + +``` +~/swh-docker$ docker-composer \ + -f base-services.yml \ + -f content-replayer-override.yml \ + config > content-replayer.yml +~/swh-docker$ docker stack deploy \ + -c content-replayer.yml \ + swh +[...] +``` + +## Full mirror + +Putting all together is just a matter of merging the compose files: + +``` +~/swh-docker$ docker-composer \ + -f base-services.yml \ + -f graph-replayer-override.yml \ + -f content-replayer-override.yml \ + config > mirror.yml +~/swh-docker$ docker stack deploy \ + -c mirror.yml \ + swh [...] ``` + ## Scaling up services In order to scale up a replayer service, you can use the `docker scale` command. For example: ``` -~/swh-docker$ docker service scale swh_graph-replayer-directory=4 +~/swh-docker$ docker service scale swh_graph-replayer=4 [...] ``` -will start 4 copies of the directory replayer service. +will start 4 copies of the graph replayer service. Notes: - One graph replayer service requires a steady 500MB to 1GB of RAM to run, so make sure you have properly sized machines for running these replayer containers, and to monitor these. - The overall bandwidth of the replayer will depend heavily on the `swh_storage` service, thus on the `swh_db-storage`. It will require some network bandwidth for the ingress kafka payload (this can easily peak to several hundreds of Mb/s). So make sure you have a correctly tuned database and enough network bw. -- Biggest topics are the directory, content and revision. +- Biggest topics are the directory, revision and content. diff --git a/base-services.yml b/base-services.yml index 8bac168..64d0b65 100644 --- a/base-services.yml +++ b/base-services.yml @@ -1,132 +1,130 @@ version: "3.7" services: memcache: image: memcached deploy: replicas: 1 db-storage: image: postgres:11 deploy: - replicas: 1 + mode: global placement: constraints: - - node.labels.role == db + - node.role == manager command: ['-c', 'shared_buffers=4GB', '-c', 'effective_cache_size=4GB', '-c', 'random_page_cost=1.5', '-c', 'max_wal_size=4GB'] environment: POSTGRES_PASSWORD_FILE: /run/secrets/postgres-password POSTGRES_USER: swh volumes: - "storage:/var/lib/postgresql/data" secrets: - source: postgres-password uid: '999' mode: 0400 web: image: softwareheritage/web:${SWH_IMAGE_TAG:-latest} configs: - source: web target: /etc/softwareheritage/config.yml - command: - - serve + command: serve environment: PORT: "5004" depends_on: - memcache objstorage: image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} deploy: - replicas: 1 placement: constraints: - node.role == manager volumes: - "objstorage:/srv/softwareheritage/objects" configs: - source: objstorage target: /etc/softwareheritage/config.yml environment: PORT: "5003" STATSD_HOST: prometheus-statsd-exporter STATSD_PORT: 9125 - command: - - objstorage + command: objstorage storage: image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} configs: - source: storage target: /etc/softwareheritage/config.yml environment: PGHOST: db-storage PORT: "5002" STATSD_HOST: prometheus-statsd-exporter STATSD_PORT: 9125 - command: - - storage + command: storage depends_on: - db-storage secrets: - source: postgres-password mode: 0400 nginx: image: nginx configs: - source: nginx target: /etc/nginx/nginx.conf ports: - "5081:5081/tcp" deploy: placement: constraints: - node.role == manager prometheus: image: prom/prometheus depends_on: - prometheus-statsd-exporter command: # Needed for the reverse-proxy - "--web.external-url=/prometheus" - "--config.file=/etc/prometheus/prometheus.yml" volumes: - "./conf/prometheus.yml:/etc/prometheus/prometheus.yml:ro" + deploy: + mode: global prometheus-statsd-exporter: image: prom/statsd-exporter command: - "--statsd.mapping-config=/etc/prometheus/statsd-mapping.yml" volumes: - "./conf/prometheus-statsd-mapping.yml:/etc/prometheus/statsd-mapping.yml:ro" grafana: image: grafana/grafana depends_on: - prometheus environment: GF_SERVER_ROOT_URL: http://localhost:5081/grafana volumes: - "./conf/grafana/provisioning:/etc/grafana/provisioning:ro" - "./conf/grafana/dashboards:/var/lib/grafana/dashboards" volumes: objstorage: storage: secrets: postgres-password: external: true configs: web: file: conf/web.yml storage: file: conf/storage.yml objstorage: file: conf/objstorage.yml nginx: file: conf/nginx.conf diff --git a/conf/content-replayer.yml.example b/conf/content-replayer.yml.example index 1d7291b..f6de22b 100644 --- a/conf/content-replayer.yml.example +++ b/conf/content-replayer.yml.example @@ -1,20 +1,24 @@ objstorage_src: cls: remote args: url: http://storage0.euwest.azure.internal.softwareheritage.org:5003 max_retries: 5 pool_connections: 100 pool_maxsize: 200 objstorage_dst: cls: remote args: url: http://objstorage:5003 -journal: +journal_client: + cls: kafka brokers: - kafka1 - kafka2 - kafka3 group_id: test-content-replayer-x-change-me - max_poll_records: 1000 + # sasl.username: XXX + # sasl.password: XXX + # security.protocol: sasl_ssl + # sasl.mechanism: SCRAM-SHA-512 diff --git a/conf/graph-replayer.yml.example b/conf/graph-replayer.yml.example index e4cea98..3767510 100644 --- a/conf/graph-replayer.yml.example +++ b/conf/graph-replayer.yml.example @@ -1,15 +1,29 @@ storage: cls: remote args: url: http://storage:5002/ max_retries: 5 pool_connections: 100 pool_maxsize: 200 -journal: +journal_client: + cls: kafka brokers: - kafka1 - kafka2 - kafka3 group_id: test-graph-replayer-x-change-me - max_poll_records: 1000 + # sasl.username: XXX + # sasl.password: XXX + # security.protocol: sasl_ssl + # sasl.mechanism: SCRAM-SHA-512 + object_types: + - content + - skipped_content + - directory + - origin + - origin_visit + - origin_visit_status + - release + - revision + - snapshot diff --git a/conf/graph-replayer/remote/all.yml.example b/conf/graph-replayer/remote/all.yml.example deleted file mode 100644 index 4246095..0000000 --- a/conf/graph-replayer/remote/all.yml.example +++ /dev/null @@ -1,22 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - # object_types: - # - content diff --git a/conf/graph-replayer/remote/content.yml.example b/conf/graph-replayer/remote/content.yml.example deleted file mode 100644 index c79d498..0000000 --- a/conf/graph-replayer/remote/content.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - content diff --git a/conf/graph-replayer/remote/directory.yml.example b/conf/graph-replayer/remote/directory.yml.example deleted file mode 100644 index 4967436..0000000 --- a/conf/graph-replayer/remote/directory.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - directory diff --git a/conf/graph-replayer/remote/origin.yml.example b/conf/graph-replayer/remote/origin.yml.example deleted file mode 100644 index 50cb945..0000000 --- a/conf/graph-replayer/remote/origin.yml.example +++ /dev/null @@ -1,24 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - origin - - origin_visit diff --git a/conf/graph-replayer/remote/other.yml.example b/conf/graph-replayer/remote/other.yml.example deleted file mode 100644 index 96f1149..0000000 --- a/conf/graph-replayer/remote/other.yml.example +++ /dev/null @@ -1,26 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - origin - - origin_visit - - snapshot - - release diff --git a/conf/graph-replayer/remote/release.yml.example b/conf/graph-replayer/remote/release.yml.example deleted file mode 100644 index 6400f74..0000000 --- a/conf/graph-replayer/remote/release.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - release diff --git a/conf/graph-replayer/remote/revision.yml.example b/conf/graph-replayer/remote/revision.yml.example deleted file mode 100644 index af56f6b..0000000 --- a/conf/graph-replayer/remote/revision.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - revision diff --git a/conf/graph-replayer/remote/snapshot.yml.example b/conf/graph-replayer/remote/snapshot.yml.example deleted file mode 100644 index b6f58cc..0000000 --- a/conf/graph-replayer/remote/snapshot.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -storage: - cls: remote - args: - url: http://storage:5002/ - pool_connections: 100 - pool_maxsize: 200 - -journal_client: - cls: kafka - brokers: - - kafka01.softwareheritage.org:9093 - - kafka02.softwareheritage.org:9093 - - kafka03.softwareheritage.org:9093 - - kafka04.softwareheritage.org:9093 - - kafka05.softwareheritage.org:9093 - - kafka06.softwareheritage.org:9093 - group_id: test-graph-replayer-XXX - sasl.username: XXX - sasl.password: XXX - security.protocol: sasl_ssl - sasl.mechanism: SCRAM-SHA-512 - object_types: - - snapshot diff --git a/content-replayer-override.yml b/content-replayer-override.yml new file mode 100644 index 0000000..01b387a --- /dev/null +++ b/content-replayer-override.yml @@ -0,0 +1,25 @@ +# Mirror specific services +version: '3.7' + +services: + storage: + command: + - fix-mirror + - storage + + content-replayer: + image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} + environment: + STATSD_HOST: prometheus-statsd-exporter + STATSD_PORT: 9125 + configs: + - source: content-replayer + target: /etc/softwareheritage/config.yml + command: + - content-replayer + depends_on: + - objstorage + +configs: + content-replayer: + file: conf/content-replayer.yml diff --git a/docker-compose-mirror.yml b/docker-compose-mirror.yml deleted file mode 100644 index afdc9c9..0000000 --- a/docker-compose-mirror.yml +++ /dev/null @@ -1,70 +0,0 @@ -# Mirror specific services -version: '3.7' - -services: - graph-replayer-content: - image: softwareheritage/base:latest - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-content - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - - graph-replayer-directory: - image: softwareheritage/base:latest - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-directory - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - - graph-replayer-revision: - image: softwareheritage/base:latest - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-revision - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - - graph-replayer-origin: - image: softwareheritage/base:latest - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-origin - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - - content-replayer: - image: softwareheritage/base:latest - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: content-replayer - target: /etc/softwareheritage/config.yml - command: - - content-replayer - -configs: - graph-replayer-directory: - file: conf/graph-replayer-directory.yml - graph-replayer-revision: - file: conf/graph-replayer-revision.yml - graph-replayer-origin: - file: conf/graph-replayer-origin.yml - graph-replayer-content: - file: conf/graph-replayer-content.yml - content-replayer: - file: conf/content-replayer.yml diff --git a/graph-replayer-override.yml b/graph-replayer-override.yml new file mode 100644 index 0000000..afdf77c --- /dev/null +++ b/graph-replayer-override.yml @@ -0,0 +1,25 @@ +# Mirror specific services +version: '3.7' + +services: + storage: + command: + - fix-mirror + - storage + + graph-replayer: + image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} + environment: + STATSD_HOST: prometheus-statsd-exporter + STATSD_PORT: 9125 + configs: + - source: graph-replayer + target: /etc/softwareheritage/config.yml + command: + - graph-replayer + depends_on: + - storage + +configs: + graph-replayer: + file: conf/graph-replayer.yml diff --git a/graph-replayer-remote-bytopic.yml b/graph-replayer-remote-bytopic.yml deleted file mode 100644 index 4e77af0..0000000 --- a/graph-replayer-remote-bytopic.yml +++ /dev/null @@ -1,95 +0,0 @@ -# Mirror specific services -version: '3.7' - -services: - graph-replayer-content: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-content - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - - graph-replayer-revision: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-revision - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - - graph-replayer-directory: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-directory - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - - graph-replayer-origin: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-origin - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - - graph-replayer-snapshot: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-snapshot - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - - graph-replayer-release: - image: softwareheritage/base:${SWH_IMAGE_TAG:-latest} - environment: - STATSD_HOST: prometheus-statsd-exporter - STATSD_PORT: 9125 - configs: - - source: graph-replayer-remote-release - target: /etc/softwareheritage/config.yml - command: - - graph-replayer - depends_on: - - storage - -configs: - graph-replayer-remote-content: - file: conf/graph-replayer/remote/content.yml - graph-replayer-remote-revision: - file: conf/graph-replayer/remote/revision.yml - graph-replayer-remote-directory: - file: conf/graph-replayer/remote/directory.yml - graph-replayer-remote-origin: - file: conf/graph-replayer/remote/origin.yml - graph-replayer-remote-release: - file: conf/graph-replayer/remote/release.yml - graph-replayer-remote-snapshot: - file: conf/graph-replayer/remote/snapshot.yml