Differential D1086 Diff 3510 docs/developer-setup.rst

Changeset View

Standalone View

docs/developer-setup.rst

This file was added.

				.. _developer-setup:

				Developer setup
				===============

				In this guide, we will set up a dual environment:

				- A virtual env in which all the \|swh\| packages will be installed in 'develop'
				mode, this will allow you to navigate the source code, hack it, and run
				locally the unit tests.

				- A docker 'cluster' built with docker-compose, which allows to easily run all
				the components of the \|swh\| architecture. It is possible to run those docker
				containers with your locally modified code for one or several \|swh\| packages.
				ardumontUnsubmitted Not Done Inline Actions containers ardumont: containers

				Please read the `README file`_ in the swh-docker-dev repository for more
				details on how to do this.

				.. _`README file`: https://forge.softwareheritage.org/source/swh-docker-dev/browse/master/README.md

				Checkout the source code
				------------------------

				Clone the \|swh\| environment repository::

				~$ git clone https://forge.softwareheritage.org/source/swh-environment.git
				[...]
				~$ cd swh-environment
				~/swh-environment$
				vlorentzUnsubmitted Not Done Inline Actions indent vlorentz: indent

				Create a virtual env::

				~/swh-environment$ mkvirtualenv -p /usr/bin/python3 -a $PWD swh
				[...]
				(swh) ~/swh-environment$


				.. Note: using virtualenvwrapper_ is not mandatory here. You can use plain
				virtualenvs, or any other venv management tool (pipenv_ or poetry_
				for example). Using a tool such as virtualenvwrapper_ just makes life
				easier...


				.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.io/
				.. _poetry: https://poetry.eustace.io/
				.. _pipenv: https://pipenv.readthedocs.io/


				Install all the swh packages (in develop mode)::

				(swh) ~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing) \
				ardumontUnsubmitted Not Done Inline Actions --with-testing? ardumont: --with-testing?
				ardumontUnsubmitted Not Done Inline Actions Also we'd like to install tox and pifpaf for running tests. pip install $(./bin/pip-swh-packages --with-testing) tox pifpaf Note: i needed to ask for wheel in the .venv, now i see it's already done by virtualenvwrapper! cool. ardumont: Also we'd like to install tox and pifpaf for running tests. ``` pip install $(./bin/pip-swh…
				tox pifpaf
				[...]


				Setup the docker environment
				----------------------------

				Install docker-compose::

				(swh) ~/swh-environment$ pip install docker-compose
				[...]

				ardumontUnsubmitted Not Done Inline Actions life ardumont: life
				Make your life easier::

				ardumontUnsubmitted Not Done Inline Actions WORKON_HOME? from virtualenvwrapper's documentation [1], it's defining the WORKON_HOME variable, not VIRTUAL_ENV. [1] https://virtualenvwrapper.readthedocs.io/en/latest/install.html#shell-startup-file ardumont: WORKON_HOME? from virtualenvwrapper's documentation [1], it's defining the WORKON_HOME variable…
				douarddaAuthorUnsubmitted Done Inline Actions Nope, WORKON_HOME is be the directory where virtualenvwrapper looks for known virtual envs. Here we want to amend the postactivate script located in the bin directory of the current virtual env, thus in $VIRTUAL_ENV/bin (VIRTUAL_ENV is defined by venv, not venvwrapper). To be really accurate, this should be $VIRTUAL_ENV/$VIRTUALENVWRAPPER_ENV_BIN_DIR but... douardda: Nope, WORKON_HOME is be the directory where virtualenvwrapper looks for known virtual envs.
				(swh) ~/swh-environment$ cat >>$VIRTUAL_ENV/bin/postactivate <<EOF
				ardumontUnsubmitted Not Done Inline Actions val is right, i'm not sure what this does. ardumont: val is right, i'm not sure what this does.
				douarddaAuthorUnsubmitted Done Inline Actions This is just the application of ckick's doc: https://click.palletsprojects.com/en/7.x/bashcomplete/#activation douardda: This is just the application of ckick's doc: https://click.palletsprojects.com/en/7.
				eval "$(_SWH_SCHEDULER_COMPLETE=source swh-scheduler)"
				export SWH_SCHEDULER_URL=http://127.0.0.1:5008/
				export CELERY_BROKER_URL=amqp://127.0.0.1:5072/
				ardumontUnsubmitted Not Done Inline Actions using an environment variable would be good here. Personally, I'm defining an SWH_ENVIRONMENT_HOME in my .sh routines (here its value would be ~/swh-environment) ardumont: using an environment variable would be good here. Personally, I'm defining an…
				douarddaAuthorUnsubmitted Done Inline Actions I do use the cdproject function to get back to this directory, so I do not really need such a variable. It could be added, but I won't add it now. douardda: I do use the cdproject function to get back to this directory, so I do not really need such a…
				export COMPOSE_FILE=~/swh-environment/swh-docker-dev/docker-compose.yml:~/swh-environment/swh-docker-dev/docker-compose.override.yml
				alias doco=docker-compose

				function swhclean {
				find ~/swh-environment -type d -name __pycache__ -exec rm -rf {} \;
				find ~/swh-environment -type d -name .tox -exec rm -rf {} \;
				find ~/swh-environment -type d -name .hypothesis -exec rm -rf {} \;
				}
				vlorentzUnsubmitted Not Done Inline Actions indent + should add a comment about what the eval does vlorentz: indent + should add a comment about what the eval does
				EOF

				This postactivate script does:

				- install a shell completion handler for the swh-scheduler command,
				ardumontUnsubmitted Not Done Inline Actions swh-scheduler ardumont: swh-scheduler
				- preset a bunch of environment variables
				ardumontUnsubmitted Not Done Inline Actions variables ardumont: variables

				- `SWH_SCHEDULER_URL` so that you can just run `sch-scheduler` against the
				scheduler API instance running in docker, without having to specify the
				endpoint URL,

				- `CELERY_BROKER` so you can execute the `celery` tool without options
				against the rabbitmq server running in the docker environment,

				- `COMPOSE_FILE` so you can run `docker-compose` from everywhere,

				- create an alias `doco` for `docker-compose` because this later is way too
				long to type,

				- add a `swhclean` shell function to clean your source directories so that
				there is no conflict with docker containers using local swh repositories (see
				below). This will delete any `.tox`, `__pycache__` and `.hypothesis`
				directory found in your swh-environment directory.


				Start the SWH platform::

				(swh) ~/swh-environment$ docker-compose up -d
				[...]

				Check celery::

				(swh) ~/swh-environment$ celery status
				listers@50ac2185c6c9: OK
				loader@b164f9055637: OK
				indexer@33bc6067a5b8: OK

				List task-types::

				(swh) ~/swh-environment$ swh-scheduler task-type list
				[...]

				Get more info on a task type::

				(swh) ~/swh-environment$ swh-scheduler task-type list -v -t origin-update-hg
				Known task types:
				origin-update-hg: swh.loader.mercurial.tasks.LoadMercurial
				Loading mercurial repository swh-loader-mercurial
				interval: 1 day, 0:00:00 [1 day, 0:00:00, 1 day, 0:00:00]
				backoff_factor: 1.0
				max_queue_length: 1000
				num_retries: None
				retry_delay: None

				Add a new task::

				(swh) ~/swh-environment$ swh-scheduler task add origin-update-hg origin_url=https://hg.logilab.org/master/cubicweb
				Created 1 tasks

				Task 1
				Next run: just now (2019-02-06 12:36:58+00:00)
				Interval: 1 day, 0:00:00
				Type: origin-update-hg
				Policy: recurring
				Args:
				Keyword args:
				origin_url: https://hg.logilab.org/master/cubicweb

				Respawn a task::

				(swh) ~/swh-environment$ swh-scheduler task respawn 1