Differential D1441 Diff 5292 docs/run_a_new_lister.rst

Changeset View

Standalone View

docs/run_a_new_lister.rst

This file was added.


				:orphan:

				.. _run-lister-tutorial:
				ardumontUnsubmitted Done Inline Actions rename this to `.. _run-lister-tutorial` You should then be able to link it ardumont: rename this to `.. _run-lister-tutorial` You should then be able to link it

				Tutorial: run a lister within docker-dev in just a few steps
				=====================================================================

				It is a good practice to run your new lister in docker-dev. This provides an almost
				ardumontUnsubmitted Done Inline Actions docker-dev. It provides an almost production-like environment. ardumont: docker-dev. It provides an almost production-like environment.
				ardumontUnsubmitted Done Inline Actions ... docker-dev. This provides ... Prefer short sentence when possible. ardumont: ``` ... docker-dev. This provides ... ``` Prefer short sentence when possible.
				production-like environment. Testing the lister in docker dev prior to deployment
				reduces the chances of encountering errors when turning it for production.
				ardumontUnsubmitted Done Inline Actions when running in production. I think we can drop `Hence ... deployment`. ardumont: when running in production. I think we can drop `Hence ... deployment`.
				Here are the steps you need to follow to run a lister within your local environment.
				ardumontUnsubmitted Done Inline Actions To run a lister within your local environment: ardumont: ``` To run a lister within your local environment: ```


				1. You must edit the docker-compose override file (`docker-compose.override.yml`).
				following the sample provided ::
				ardumontUnsubmitted Done Inline Actions Edit a `docker-compose.override.yml`, following the sample provided: ardumont: 1. Edit a `docker-compose.override.yml`, following the sample provided:

				version: '2'

				services:
				swh-lister:
				volumes:
				- "$SWH_ENVIRONMENT_HOME/swh-lister:/src/swh-lister"

				The file named `docker-compose.override.yml` will automatically be loaded by
				`docker-compose`.Having an override makes it possible to run a docker container
				with some swh packages installed from sources instead of using the latest
				ardumontUnsubmitted Done Inline Actions Please explain that this use docker with mount points on your local environment (your modified code) and not the pypi one. That allows to execute the development version. ardumont: Please explain that this use docker with mount points on your local environment (your modified…
				published packages from pypi. For more details, you may refer to README.md
				present in `swh-docker-dev`.
				ardumontUnsubmitted Done Inline Actions Highlight `Preparation steps and Configuration` ardumont: Highlight `Preparation steps and Configuration`
				2. Follow the instruction mentioned under heading Preparation steps and
				Configuration file sample in README.md of swh-lister.
				3. Make sure to run storage (5002) and scheduler (5008) services locally.
				You may use the following command to run docker::

				~/swh-environment/swh-docker-dev$ docker-compose up -d

				4. Add the lister task-type in the scheduler. For example, if you want to
				add pypi lister task-type ::

				ardumontUnsubmitted Done Inline Actions You are missing the backend name (fqdn of the lister task needed for the code to run) as 2nd parameter of the cli, please fix this to: swh scheduler task-type add list-gnu-full "swh.lister.gnu.tasks.GNUListerTask" "Full GNU lister" \ --default-interval '1 day' \ --backoff-factor 1 Note: that must be the reason of your error below. ardumont: You are missing the backend name (fqdn of the lister task needed for the code to run) as 2nd…
				nahimilegaAuthorUnsubmitted Done Inline Actions I added the task type as you recommended. (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task-type list -v ... list-gnu-full: swh.lister.gnu.tasks.GNUListerTask Full GNU lister interval: 1 day, 0:00:00 [None, None] backoff_factor: 1.0 max_queue_length: None num_retries: None retry_delay: None ... Now I created the task as swh scheduler --url http://localhost:5008/ task add \ list-gnu-full --policy oneshot But when I checked for tasks created by the lister, there were none (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task list Found 3 tasks Task 16809 Next run: 12 minutes ago (2019-06-17 19:02:11+00:00) Interval: 1 day, 0:00:00 Type: list-gnu-full Policy: oneshot Status: disabled Priority: Args: Keyword args: There is something still something wrong. The lister is not creating the loading task. Can you please help me find where the steps went wrong. nahimilega: I added the task type as you recommended. ``` (swh) archit@work-pc:~/swh-environment/swh…
				ardumontUnsubmitted Done Inline Actions Have you added the associated load-gnu task (as correctly mentioned in the doc ;)? ardumont: Have you added the associated load-gnu task (as correctly mentioned in the doc ;)?
				~/swh-environment$swh scheduler task-type add list-gnu-full \
				"swh.lister.gnu.tasks.GNUListerTask" "Full GNU lister" \
				--default-interval '1 day' --backoff-factor 1

				You can check all the task-type by::

				~/swh-environment$swh scheduler task-type list
				Known task types:
				list-bitbucket-incremental:
				Incrementally list BitBucket
				list-cran:
				Full CRAN Lister
				list-debian-distribution:
				List a Debian distribution
				list-github-full:
				Full update of GitHub repos list
				list-github-incremental:
				...

				If your lister is creating new loading task not yet registered, you need
				to register that task type as well. Like for GNU lister::

				~/swh-environment$swh scheduler task-type add load-gnu-full recurring \
				"GNU Loader"
				ardumontUnsubmitted Done Inline Actions `. You` (missing a space in between) ardumont: `. You` (missing a space in between)

				ardumontUnsubmitted Done Inline Actions scheduler ardumont: scheduler
				5. Run your lister with the help of scheduler cli. You need to add the task in
				the scheduler using its cli. For example, you need to execute this command
				to run gnu lister ::

				nahimilegaAuthorUnsubmitted Done Inline Actions I tested this method of running lister, but when I used it, it only created a task of "list-gnu-full", nothing else. No other code was executed, neither more task of "load-gnu" created. Whereas when I used in python code which was mentioned in README, it ran pretty nice and created loading tasks. Can you please tell me what could be the reason for this behaviour? nahimilega: I tested this method of running lister, but when I used it, it only created a task of "list-gnu…
				~/swh-environment$swh scheduler --url http://localhost:5008/ task add \
				list-gnu-full --policy oneshot

				After the execution of lister is complete, you can see the loading task created.
				~/swh-environment/swh-lister$swh scheduler task list
				ardumontUnsubmitted Done Inline Actions You can also add a note to check that the lister's cache (table in db) has been indeed filled. ardumont: You can also add a note to check that the lister's cache (table in db) has been indeed filled.