diff --git a/docs/index.rst b/docs/index.rst --- a/docs/index.rst +++ b/docs/index.rst @@ -19,6 +19,7 @@ tutorial develop_a_new_lister + run_lister Reference Documentation diff --git a/docs/run_lister.rst b/docs/run_lister.rst new file mode 100644 --- /dev/null +++ b/docs/run_lister.rst @@ -0,0 +1,296 @@ +This page describe step by step how to deploy and initialize a new lister on the staging +infractucture + +.. _deploy_the_new_lister_version: + +Deploy the new lister version +============================= + +.. _upgrade_the_packages: + +Upgrade the packages +-------------------- + +The actions can be launched on all the workers at the same time with clush. + +There is more information about clush on the `Upgrades +`__ page. + +With ``clush`` on ``pergamon``: + +:: + + # check current version + clush -b -w @staging-workers dpkg -l python3-swh.lister + # Upgrade + clush -b -w @staging-workers 'apt-get update; apt-get install -y python3-swh.lister' + # check the new version is well deployed + clush -b -w @staging-workers dpkg -l python3-swh.lister + +Example of execution: + +:: + + # check current version + root@pergamon:~# clush -b -w @staging-workers 'dpkg -l python3-swh.lister' + --------------- + worker[0-2].internal.staging.swh.network (3) + --------------- + Desired=Unknown/Install/Remove/Purge/Hold + | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend + |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) + ||/ Name Version Architecture Description + +++-==================-====================-============-================================================================= + ii python3-swh.lister 0.1.4-0~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade + + # Upgrade + root@pergamon:~# clush -b -w @staging-workers 'apt-get update; apt-get install -y python3-swh.lister' + ... + + # check the new version is well deployed + root@pergamon:~# clush -b -w @staging-workers dpkg -l python3-swh.lister + --------------- + worker[0-2].internal.staging.swh.network (3) + --------------- + Desired=Unknown/Install/Remove/Purge/Hold + | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend + |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) + ||/ Name Version Architecture Description + +++-==================-====================-============-================================================================= + ii python3-swh.lister 0.1.4-1~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade + +.. _upgrade_the_puppet_configuration: + +Upgrade the puppet configuration +================================ + +Each type of task is associated to a rabbitmq queue. To have the listers watching to the +new(s) queue(s), `the configuration deployed by puppet +`__ +must be updated to reference them. The exact name of the task to add is the package +name + `the name declared on the tasks +`__ +themselves. + +the puppet master must be refreshed and the configuration deployed. + +.. _upgrade_the_puppet_master: + +Upgrade the puppet master +------------------------- + +On pergamon: + +:: + + root@pergamon:~# /usr/local/bin/deploy.sh -v + +.. _apply_the_configuration_on_the_workers: + +Apply the configuration on the workers +-------------------------------------- + +Still on pergamon: + +:: + + root@pergamon:~# clush -b -w @staging-workers puppet agent -t + +.. _restart_the_listers: + +Restart the listers +=================== + +On ``pergamon``, with ``clush`` + +:: + + clush -b -w @staging-workers 'systemctl restart swh-worker@lister' + +.. _create_the_model_and_update_the_scheduler: + +Create the model and update the scheduler +========================================= + +At this stage, the listers are up to date but not yet ready to accept +new tasks. + +The listers database have to be updated to declare the new tables needed +by the new lister. The commands lines are executed on the scheduler +server by convenience as it have all the configuration to do it. + +.. _upgrade_the_scheduler_packages: + +Upgrade the scheduler packages +------------------------------ + +On the scheduler server (``scheduler0.internal.staging.swh.network`` on +staging) with the \*\ ``root``\ \* user: + +:: + + # Update the lister package to refresh the cli tool + apt-get update + apt-get install -y python3-swh.lister + # check the version is the same as the workers + dpkg -l python3-swh.lister + +.. _update_the_listers_database_model: + +Update the **listers** database model +------------------------------------- + +With the ``swhscheduler`` user: + +:: + + swh lister --db-url postgresql://swh-lister:******@db0.internal.staging.swh.network:5432/swh-lister db-init + +Example: + +:: + + swhscheduler@scheduler0:~$ swh lister --db-url postgresql://swh-lister:******@db0.internal.staging.swh.network:5432/swh-lister db-init + WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' + INFO:swh.lister.cli:Loading lister bitbucket + INFO:swh.lister.cli:Loading lister cgit + INFO:swh.lister.cli:Loading lister cran + INFO:swh.lister.cli:Loading lister debian + INFO:swh.lister.cli:Loading lister gitea + INFO:swh.lister.cli:Loading lister github + INFO:swh.lister.cli:Loading lister gitlab + INFO:swh.lister.cli:Loading lister gnu + INFO:swh.lister.cli:Loading lister launchpad + INFO:swh.lister.cli:Loading lister npm + INFO:swh.lister.cli:Loading lister packagist + INFO:swh.lister.cli:Loading lister phabricator + INFO:swh.lister.cli:Loading lister pypi + INFO:swh.lister.cli:Initializing database + INFO:swh.lister.core.models:Creating tables + INFO:swh.lister.cli:Calling init hook for debian + +As a last step, we need to grant read access to user guest to the new +table: + +| ``   grant select`` +| ``   on all tables in schema public`` +| ``   to guest;`` + +.. _configure_the_scheduler: + +Configure the scheduler +----------------------- + +- Check the task type is well present in the plugin list of the swh + command line with: + +:: + + swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register --help + +Example: + +:: + + swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register --help + ... + + Options: + -p, --plugins [all|loader.archive|loader.cran|loader.debian|loader.deposit|loader.nixguix|loader.npm|loader.pypi|lister.bitbucket|lister.cgit|lister.cran|lister.debian|lister.gitea|lister.github|lister.gitlab|lister.gnu|lister.launchpad|lister.npm|lister.packagist|lister.phabricator|lister.pypi] + Registers task-types for provided plugins. + Defaults to all + ... + +- Register the task type in the scheduler: + +On the scheduler server, with the ``swhscheduler`` user + +:: + + swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p + +Example: + +:: + + swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad + WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' + INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml + INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad + INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler + INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler + INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler + +(!) The naming of the task must be list-- + +.. raw:: html + + + +, if not the code must be adapted accordingly. + +.. _register_a_new_task: + +Register a new task +------------------- + +The listers and the scheduler are now ready to work together. + +To test, a task can be created in the scheduler. To do so, on the +scheduler server, with the ``swhscheduler`` user: + +:: + + swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add [--policy [recurring|oneshot]] [param1=value1] [param2=value2] + +For examples: + +- To add a task requiring no parameters (launchpad lister) + +:: + + swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add list-launchpad-full + WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' + INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml + Created 1 tasks + + Task 1240540 + Next run: just now (2020-09-08 13:08:07+00:00) + Interval: 90 days, 0:00:00 + Type: list-launchpad-full + Policy: recurring + Args: + Keyword args: + +- To add a one-shot task with parameters: + +:: + + swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add --policy oneshot list-gitea-full url=https://codeberg.org/api/v1/ limit=100 + WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal' + INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml + Created 1 tasks + + Task 1240540 + Next run: just now (2020-09-11 14:25:45+00:00) + Interval: 90 days, 0:00:00 + Type: list-gitea-full + Policy: oneshot + Args: + Keyword args: + limit: 100 + url: 'https://codeberg.org/api/v1/' + +.. _check_the_logs: + +Check the logs +-------------- + +On ``pergamon`` with ``root`` (or as a sudo user): + +:: + + clush -b -w @staging-workers 'systemctl status swh-worker@lister' + +It will output the status of the listers and the last lines of the +lister's logs on each worker server.