Differential D1441 Diff 5380 docs/run_a_new_lister.rst

Changeset View

Standalone View

docs/run_a_new_lister.rst

This file was added.


				:orphan:

				.. _run-lister-tutorial:
				ardumontUnsubmitted Done Inline Actions rename this to `.. _run-lister-tutorial` You should then be able to link it ardumont: rename this to `.. _run-lister-tutorial` You should then be able to link it

				Tutorial: run a lister within docker-dev in just a few steps
				=====================================================================

				It is a good practice to run your new lister in docker-dev. This provides an almost
				ardumontUnsubmitted Done Inline Actions docker-dev. It provides an almost production-like environment. ardumont: docker-dev. It provides an almost production-like environment.
				ardumontUnsubmitted Done Inline Actions ... docker-dev. This provides ... Prefer short sentence when possible. ardumont: ``` ... docker-dev. This provides ... ``` Prefer short sentence when possible.
				production-like environment. Testing the lister in docker dev prior to deployment
				reduces the chances of encountering errors when turning it for production.
				ardumontUnsubmitted Done Inline Actions when running in production. I think we can drop `Hence ... deployment`. ardumont: when running in production. I think we can drop `Hence ... deployment`.
				Here are the steps you need to follow to run a lister within your local environment.
				ardumontUnsubmitted Done Inline Actions To run a lister within your local environment: ardumont: ``` To run a lister within your local environment: ```


				1. You must edit the docker-compose override file (`docker-compose.override.yml`).
				following the sample provided ::
				ardumontUnsubmitted Done Inline Actions Edit a `docker-compose.override.yml`, following the sample provided: ardumont: 1. Edit a `docker-compose.override.yml`, following the sample provided:

				version: '2'

				services:
				swh-lister:
				volumes:
				- "$SWH_ENVIRONMENT_HOME/swh-lister:/src/swh-lister"

				The file named `docker-compose.override.yml` will automatically be loaded by
				``docker-compose``.Having an override makes it possible to run a docker container
				with some swh packages installed from sources instead of using the latest
				ardumontUnsubmitted Done Inline Actions Please explain that this use docker with mount points on your local environment (your modified code) and not the pypi one. That allows to execute the development version. ardumont: Please explain that this use docker with mount points on your local environment (your modified…
				published packages from pypi. For more details, you may refer to README.md
				present in ``swh-docker-dev``.
				ardumontUnsubmitted Done Inline Actions Highlight `Preparation steps and Configuration` ardumont: Highlight `Preparation steps and Configuration`
				2. Follow the instruction mentioned under heading Preparation steps and
				Configuration file sample in README.md of swh-lister.
				3. Add in the lister configuration the new ``task_modules`` and ``task_queues``
				entry for the your new lister. You need to amend the conf/lister.yml file to
				add the entries. Here is an example for GNU lister::

				celery:
				task_broker: amqp://guest:guest@amqp//
				task_modules:
				...
				ardumontUnsubmitted Done Inline Actions You are missing the backend name (fqdn of the lister task needed for the code to run) as 2nd parameter of the cli, please fix this to: swh scheduler task-type add list-gnu-full "swh.lister.gnu.tasks.GNUListerTask" "Full GNU lister" \ --default-interval '1 day' \ --backoff-factor 1 Note: that must be the reason of your error below. ardumont: You are missing the backend name (fqdn of the lister task needed for the code to run) as 2nd…
				nahimilegaAuthorUnsubmitted Done Inline Actions I added the task type as you recommended. (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task-type list -v ... list-gnu-full: swh.lister.gnu.tasks.GNUListerTask Full GNU lister interval: 1 day, 0:00:00 [None, None] backoff_factor: 1.0 max_queue_length: None num_retries: None retry_delay: None ... Now I created the task as swh scheduler --url http://localhost:5008/ task add \ list-gnu-full --policy oneshot But when I checked for tasks created by the lister, there were none (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task list Found 3 tasks Task 16809 Next run: 12 minutes ago (2019-06-17 19:02:11+00:00) Interval: 1 day, 0:00:00 Type: list-gnu-full Policy: oneshot Status: disabled Priority: Args: Keyword args: There is something still something wrong. The lister is not creating the loading task. Can you please help me find where the steps went wrong. nahimilega: I added the task type as you recommended. ``` (swh) archit@work-pc:~/swh-environment/swh…
				ardumontUnsubmitted Done Inline Actions Have you added the associated load-gnu task (as correctly mentioned in the doc ;)? ardumont: Have you added the associated load-gnu task (as correctly mentioned in the doc ;)?
				- swh.lister.gnu.tasks
				task_queues:
				...
				- swh.lister.gnu.tasks.GNUListerTask

				4. Make sure to run ``storage (5002)`` and ``scheduler (5008)`` services locally.
				You may use the following command to run docker::

				~/swh-environment/swh-docker-dev$ docker-compose up -d

				5. Add the lister task-type in the scheduler. For example, if you want to
				add pypi lister task-type ::

				~/swh-environment$ swh scheduler task-type add list-gnu-full \
				"swh.lister.gnu.tasks.GNUListerTask" "Full GNU lister" \
				--default-interval '1 day' --backoff-factor 1

				You can check all the task-type by::

				~/swh-environment$swh scheduler task-type list
				Known task types:
				list-bitbucket-incremental:
				Incrementally list BitBucket
				list-cran:
				ardumontUnsubmitted Done Inline Actions `. You` (missing a space in between) ardumont: `. You` (missing a space in between)
				Full CRAN Lister
				ardumontUnsubmitted Done Inline Actions scheduler ardumont: scheduler
				list-debian-distribution:
				List a Debian distribution
				list-github-full:
				Full update of GitHub repos list
				nahimilegaAuthorUnsubmitted Done Inline Actions I tested this method of running lister, but when I used it, it only created a task of "list-gnu-full", nothing else. No other code was executed, neither more task of "load-gnu" created. Whereas when I used in python code which was mentioned in README, it ran pretty nice and created loading tasks. Can you please tell me what could be the reason for this behaviour? nahimilega: I tested this method of running lister, but when I used it, it only created a task of "list-gnu…
				list-github-incremental:
				...

				If your lister is creating new loading task not yet registered, you need
				to register that task type as well.
				ardumontUnsubmitted Done Inline Actions You can also add a note to check that the lister's cache (table in db) has been indeed filled. ardumont: You can also add a note to check that the lister's cache (table in db) has been indeed filled.

				6. Run your lister with the help of scheduler cli. You need to add the task in
				the scheduler using its cli. For example, you need to execute this command
				to run gnu lister ::

				~/swh-environment$ swh scheduler --url http://localhost:5008/ task add \
				list-gnu-full --policy oneshot

				After the execution of lister is complete, you can see the loading task created::

				~/swh-environment/swh-lister$ swh scheduler task list

				You can also check the repositories listed by the lister from the database in
				which the lister output is stored. To connect to the database::

				~/swh-environment/swh-docker-dev$ docker-compose exec swh-lister bash -c \
				'psql swh-listers'