Changeset View
Changeset View
Standalone View
Standalone View
docker/README.md
Show First 20 Lines • Show All 73 Lines • ▼ Show 20 Lines | |||||
Attaching to docker_swh-lister_1 | Attaching to docker_swh-lister_1 | ||||
[...] | [...] | ||||
swh-lister_1 | Processing /src/swh-scheduler | swh-lister_1 | Processing /src/swh-scheduler | ||||
swh-lister_1 | Could not install packages due to an EnvironmentError: [('/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz', '/tmp/pip-req-build-pm7nsax3/.hypothesis/unicodedata/8.0.0/charmap.json.gz', "[Errno 13] Permission denied: '/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz'")] | swh-lister_1 | Could not install packages due to an EnvironmentError: [('/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz', '/tmp/pip-req-build-pm7nsax3/.hypothesis/unicodedata/8.0.0/charmap.json.gz', "[Errno 13] Permission denied: '/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz'")] | ||||
swh-lister_1 | | swh-lister_1 | | ||||
``` | ``` | ||||
Once all containers are running, you can use the web interface by opening | Once all containers are running, you can use the web interface by opening | ||||
http://localhost:5080/ in your web browser. | http://localhost:5081/ in your web browser. | ||||
At this point, the archive is empty and needs to be filled with some content. | At this point, the archive is empty and needs to be filled with some content. | ||||
To do so, you can create tasks that will scrape a forge. For example, to inject | To do so, you can create tasks that will scrape a forge. For example, to inject | ||||
the code from the https://0xacab.org gitlab forge: | the code from the https://0xacab.org gitlab forge: | ||||
``` | ``` | ||||
~/swh-environment/docker$ docker-compose exec swh-scheduler \ | ~/swh-environment/docker$ docker-compose exec swh-scheduler \ | ||||
swh scheduler task add list-gitlab-full \ | swh scheduler task add list-gitlab-full \ | ||||
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 Lines | |||||
### Exposed Ports | ### Exposed Ports | ||||
Several services have their listening ports exposed on the host: | Several services have their listening ports exposed on the host: | ||||
- amqp: 5072 | - amqp: 5072 | ||||
- kafka: 5092 | - kafka: 5092 | ||||
- nginx: 5080 | - nginx: 5081 | ||||
And for SWH services: | And for SWH services: | ||||
- scheduler API: 5008 | - scheduler API: 5008 | ||||
- storage API: 5002 | - storage API: 5002 | ||||
- object storage API: 5003 | - object storage API: 5003 | ||||
- indexer API: 5007 | - indexer API: 5007 | ||||
- web app: 5004 | - web app: 5004 | ||||
▲ Show 20 Lines • Show All 147 Lines • ▼ Show 20 Lines | index-origin-metadata: | ||||
Origin Metadata indexer task | Origin Metadata indexer task | ||||
``` | ``` | ||||
### Monitoring activity | ### Monitoring activity | ||||
You can monitor the workers activity by connecting to the RabbitMQ console on | You can monitor the workers activity by connecting to the RabbitMQ console on | ||||
`http://localhost:5080/rabbitmq` or the grafana dashboard on | `http://localhost:5081/rabbitmq` or the grafana dashboard on | ||||
`http://localhost:5080/grafana`. | `http://localhost:5081/grafana`. | ||||
If you cannot see any task being executed, check the logs of the | If you cannot see any task being executed, check the logs of the | ||||
`swh-scheduler-runner` service (here is a failure example due to the | `swh-scheduler-runner` service (here is a failure example due to the | ||||
debian lister task not being properly registered on the | debian lister task not being properly registered on the | ||||
swh-scheduler-runner service): | swh-scheduler-runner service): | ||||
``` | ``` | ||||
~/swh-environment/docker$ docker-compose logs --tail=10 swh-scheduler-runner | ~/swh-environment/docker$ docker-compose logs --tail=10 swh-scheduler-runner | ||||
▲ Show 20 Lines • Show All 309 Lines • Show Last 20 Lines |