Changeset View
Changeset View
Standalone View
Standalone View
docker/README.rst
Show First 20 Lines • Show All 577 Lines • ▼ Show 20 Lines | |||||
This way, ``docker-compose down`` without the ``-v`` flag will not | This way, ``docker-compose down`` without the ``-v`` flag will not | ||||
remove those volumes and data will persist. | remove those volumes and data will persist. | ||||
Starting a kafka-powered mirror of the storage | Starting a kafka-powered mirror of the storage | ||||
---------------------------------------------- | ---------------------------------------------- | ||||
This repo comes with an optional ``docker-compose.storage-mirror.yml`` | This repo comes with an optional ``docker-compose.storage-mirror.yml`` | ||||
docker compose file that can be used to test the kafka-powered mirror | docker compose file that can be used to test the kafka-powered mirror | ||||
mecanism for the main storage. | mechanism for the main storage. | ||||
This can be used like:: | This can be used like:: | ||||
~/swh-environment/docker$ docker-compose -f docker-compose.yml -f docker-compose.storage-mirror.yml up -d | ~/swh-environment/docker$ docker-compose -f docker-compose.yml -f docker-compose.storage-mirror.yml up -d | ||||
[...] | [...] | ||||
Compared to the original compose file, this will: | Compared to the original compose file, this will: | ||||
- overrides the swh-storage service to activate the kafka direct writer | - overrides the swh-storage service to activate the kafka direct writer | ||||
on swh.journal.objects prefixed topics using thw swh.storage.master | on swh.journal.objects prefixed topics using the swh.storage.master | ||||
ID, | ID, | ||||
- overrides the swh-web service to make it use the mirror instead of | - overrides the swh-web service to make it use the mirror instead of | ||||
the master storage, | the master storage, | ||||
- starts a db for the mirror, | - starts a db for the mirror, | ||||
- starts a storage service based on this db, | - starts a storage service based on this db, | ||||
- starts a replayer service that runs the process that listen to kafka | - starts a replayer service that runs the process that listen to kafka | ||||
to keeps the mirror in sync. | to keeps the mirror in sync. | ||||
Show All 23 Lines | (swh)$ docker-compose \ | ||||
--start-object 000000 \ | --start-object 000000 \ | ||||
--end-object 000001 \ | --end-object 000001 \ | ||||
--dry-run | --dry-run | ||||
Using Sentry | Using Sentry | ||||
------------ | ------------ | ||||
All entrypoints to SWH code (CLI, gunicorn, celery, …) are, or should | All entrypoints to SWH code (CLI, gunicorn, celery, …) are, or should | ||||
be, intrumented using Sentry. By default this is disabled, but if you | be, instrumented using Sentry. By default this is disabled, but if you | ||||
run your own Sentry instance, you can use it. | run your own Sentry instance, you can use it. | ||||
To do so, you must get a DSN from your Sentry instance, and set it as | To do so, you must get a DSN from your Sentry instance, and set it as | ||||
the value of ``SWH_SENTRY_DSN`` in the file ``env/common_python.env``. | the value of ``SWH_SENTRY_DSN`` in the file ``env/common_python.env``. | ||||
You may also set it per-service in the ``environment`` section of each | You may also set it per-service in the ``environment`` section of each | ||||
services in ``docker-compose.override.yml``. | services in ``docker-compose.override.yml``. | ||||
Caveats | Caveats | ||||
------- | ------- | ||||
Running a lister task can lead to a lot of loading tasks, which can fill | Running a lister task can lead to a lot of loading tasks, which can fill | ||||
your hard drive pretty fast. Make sure to monitor your available storage | your hard drive pretty fast. Make sure to monitor your available storage | ||||
space regularly when playing with this stack. | space regularly when playing with this stack. | ||||
Also, a few containers (``swh-storage``, ``swh-xxx-db``) use a volume | Also, a few containers (``swh-storage``, ``swh-xxx-db``) use a volume | ||||
for storing the blobs or the database files. With the default | for storing the blobs or the database files. With the default | ||||
configuration provided in the ``docker-compose.yml`` file, these volumes | configuration provided in the ``docker-compose.yml`` file, these volumes | ||||
are not persistant. So removing the containers will delete the volumes! | are not persistent. So removing the containers will delete the volumes! | ||||
Also note that for the ``swh-objstorage``, since the volume can be | Also note that for the ``swh-objstorage``, since the volume can be | ||||
pretty big, the remove operation can be quite long (several minutes is | pretty big, the remove operation can be quite long (several minutes is | ||||
not uncommon), which may mess a bit with the ``docker-compose`` command. | not uncommon), which may mess a bit with the ``docker-compose`` command. | ||||
If you have an error message like: | If you have an error message like: | ||||
Error response from daemon: removal of container 928de3110381 is already | Error response from daemon: removal of container 928de3110381 is already | ||||
in progress | in progress | ||||
it means that you need to wait for this process to finish before being | it means that you need to wait for this process to finish before being | ||||
able to (re)start your docker stack again. | able to (re)start your docker stack again. |