Changeset View
Changeset View
Standalone View
Standalone View
README.md
# swh-docker-dev | # swh-docker-dev | ||||
[Work in progress] | [Work in progress] | ||||
This repo contains Dockerfiles to allow developers to run a small | This repo contains Dockerfiles to allow developers to run a small | ||||
Software Heritage instance on their development computer. | Software Heritage instance on their development computer. | ||||
The end goal is to smooth the contributors/developers workflow. Focus | The end goal is to smooth the contributors/developers workflow. Focus | ||||
on coding, not configuring! | on coding, not configuring! | ||||
## Dependencies | ## Dependencies | ||||
This uses docker with docker-compose, so ensure you have a working | This uses docker with docker-compose, so ensure you have a working | ||||
docker environment and docker-compose is installed. | docker environment and docker-compose is installed. | ||||
## Warning | |||||
Running a Software Heritage instance on your machine can be quickly quite | |||||
ardumont: resource | |||||
ressource consuming: if you play a bit too hard (eg. if you try the github | |||||
lister), you may fill your hard drive pretty quick, and consume a lot of CPU, | |||||
memory and network bandwidth. | |||||
## Quick start | ## Quick start | ||||
First, start containers: | First, start containers: | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ docker-compose up -d | ~/swh-environment/swh-docker-dev$ docker-compose up -d | ||||
[...] | [...] | ||||
Creating swh-docker-dev_amqp_1 ... done | Creating swh-docker-dev_amqp_1 ... done | ||||
Show All 31 Lines | |||||
swh-lister-debian_1 | Processing /src/swh-scheduler | swh-lister-debian_1 | Processing /src/swh-scheduler | ||||
swh-lister-debian_1 | Could not install packages due to an EnvironmentError: [('/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz', '/tmp/pip-req-build-pm7nsax3/.hypothesis/unicodedata/8.0.0/charmap.json.gz', "[Errno 13] Permission denied: '/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz'")] | swh-lister-debian_1 | Could not install packages due to an EnvironmentError: [('/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz', '/tmp/pip-req-build-pm7nsax3/.hypothesis/unicodedata/8.0.0/charmap.json.gz', "[Errno 13] Permission denied: '/src/swh-scheduler/.hypothesis/unicodedata/8.0.0/charmap.json.gz'")] | ||||
swh-lister-debian_1 | | swh-lister-debian_1 | | ||||
``` | ``` | ||||
Once all the containers are running, you can use the web interface by opening | Once all the containers are running, you can use the web interface by opening | ||||
http://localhost:5080/ in your web browser. | http://localhost:5080/ in your web browser. | ||||
At this point, the archive is empty and needs to be filled with content. To do | At this point, the archive is empty and needs to be filled with some content. | ||||
so, you can create tasks that will scrape a forge. For example, to inject the | To do so, you can create tasks that will scrape a forge. For example, to inject | ||||
code from the https://0xacab.org gitlab forge: | the code from the https://0xacab.org gitlab forge: | ||||
Not Done Inline Actionscontents? ardumont: contents? | |||||
``` | ``` | ||||
$ ~/swh-environment/swh-docker-dev$ docker-compose run swh-scheduler-api \ | $ ~/swh-environment/swh-docker-dev$ docker-compose run swh-scheduler-api \ | ||||
swh-scheduler -c remote -u http://swh-scheduler-api:5008/ \ | swh-scheduler -c remote -u http://swh-scheduler-api:5008/ \ | ||||
task add swh-lister-gitlab-full -p oneshot api_baseurl=https://0xacab.org/api/v4 | task add swh-lister-gitlab-full -p oneshot api_baseurl=https://0xacab.org/api/v4 | ||||
Created 1 tasks | Created 1 tasks | ||||
Task 1 | Task 1 | ||||
Next run: just now (2018-12-19 14:58:49+00:00) | Next run: just now (2018-12-19 14:58:49+00:00) | ||||
Interval: 90 days, 0:00:00 | Interval: 90 days, 0:00:00 | ||||
Type: swh-lister-gitlab-full | Type: swh-lister-gitlab-full | ||||
Policy: oneshot | Policy: oneshot | ||||
Args: | Args: | ||||
Keyword args: | Keyword args: | ||||
api_baseurl=https://0xacab.org/api/v4 | api_baseurl=https://0xacab.org/api/v4 | ||||
``` | ``` | ||||
This task will scrape the forge's project list and create subtasks to inject | This task will scrape the forge's project list and create subtasks to inject | ||||
each git repository found there. | each git repository found there. | ||||
This will take a bit af time to complete, but you can follow Celery activity on | This will take a bit af time to complete. | ||||
the flower instance: http://localhost:5080/flower/ | |||||
To increase the speed at wich git repositories are imported, you can spawn more | To increase the speed at wich git repositories are imported, you can spawn more | ||||
`swh-loader-git` workers: | `swh-loader-git` workers: | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ export CELERY_BROKER_URL=amqp://:5072// | ~/swh-environment/swh-docker-dev$ export CELERY_BROKER_URL=amqp://:5072// | ||||
~/swh-environment/swh-docker-dev$ celery status | ~/swh-environment/swh-docker-dev$ celery status | ||||
mercurial@8f63da914c26: OK | mercurial@8f63da914c26: OK | ||||
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines | |||||
`docker-compose`. | `docker-compose`. | ||||
This example shows the simple case of the `swh-objstorage` package: you just have to | This example shows the simple case of the `swh-objstorage` package: you just have to | ||||
mount it in the container in `/src` and the entrypoint will ensure every | mount it in the container in `/src` and the entrypoint will ensure every | ||||
swh-* package found in `/src/` is installed (using `pip install -e` so you can | swh-* package found in `/src/` is installed (using `pip install -e` so you can | ||||
easily hack your code. If the application you play with have autoreload support, | easily hack your code. If the application you play with have autoreload support, | ||||
there is even no need for restarting the impacted container.) | there is even no need for restarting the impacted container.) | ||||
Note: if the docker fails to start when using local sources for one or more swh | |||||
Not Done Inline Actionspackages, ... ardumont: packages, ... | |||||
package, it's most probably due to permission problems on cache files. For | |||||
example, if you have executed tests locally (using pytest or tox), you have | |||||
cache files (__pycache__ etc.) that will prevent `pip install` from working | |||||
within the docker. | |||||
The solution is to clean these files and directories before trying to spawn the | |||||
docker. | |||||
``` | |||||
~/swh-environment$ find . -type d -name __pycache__ -exec rm -rf {} \; | |||||
~/swh-environment$ find . -type d -name .tox -exec rm -rf {} \; | |||||
Not Done Inline Actionsswhclean? you mentioned and installed it on the post hook... ardumont: swhclean?
you mentioned and installed it on the post hook...
granted, not everyone could use… | |||||
~/swh-environment$ find . -type d -name .hypothesis -exec rm -rf {} \; | |||||
``` | |||||
## Details | ## Details | ||||
This runs the following services on their respectively standard ports, | This runs the following services on their respectively standard ports, | ||||
all of the following services are configured to communicate with each | all of the following services are configured to communicate with each | ||||
other: | other: | ||||
- swh-storage-db: a `softwareheritage` instance db that stores the | - swh-storage-db: a `softwareheritage` instance db that stores the | ||||
Merkle DAG, | Merkle DAG, | ||||
▲ Show 20 Lines • Show All 219 Lines • Show Last 20 Lines |
resource