diff --git a/sysadm/deployment/deploy-lister.rst b/sysadm/deployment/deploy-lister.rst index c9a5654..2c6e41d 100644 --- a/sysadm/deployment/deploy-lister.rst +++ b/sysadm/deployment/deploy-lister.rst @@ -1,166 +1,166 @@ .. _deploy-new-lister: How to deploy a new lister ========================== This page describes step by step how to deploy and initialize a new lister on the staging infractucture. .. _upgrade-the-packages: Upgrade the packages -------------------- The actions can be launched on all the workers at the same time with ``clush``. There is more information about clush on the `Upgrades `__ page. With ``clush`` on ``pergamon``: :: # check current version clush -b -w @staging-loader-workers dpkg -l python3-swh.lister # Upgrade clush -b -w @staging-loader-workers 'apt-get update; apt-get install -y python3-swh.lister' # check the new version is well deployed clush -b -w @staging-loader-workers dpkg -l python3-swh.lister Example of execution: :: # check current version root@pergamon:~# clush -b -w @staging-loader-workers 'dpkg -l python3-swh.lister | grep ii' ii python3-swh.lister 0.1.4-0~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade # Upgrade root@pergamon:~# clush -b -w @staging-loader-workers 'apt-get update; apt-get install -y python3-swh.lister' ... # check the new version is well deployed root@pergamon:~# clush -b -w @staging-loader-workers "dpkg -l python3-swh.lister | grep ii" ii python3-swh.lister 0.1.4-1~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)# Upgrade .. _upgrade_the_puppet_configuration: Upgrade the puppet configuration -------------------------------- Each type of task is associated to a rabbitmq queue. To have the listers watching to the -new(s) queue(s), `the configuration deployed by puppet -`__ +new(s) queue(s), :swh_web:`the configuration deployed by puppet +` must be updated to reference them. -The exact name of the task to add is the package name and `the name declared on the +The exact name of the task to add is the package name and :swh_web:`the name declared on the tasks -`__ +` themselves. The puppet master must be refreshed and the configuration deployed. .. _upgrade-the-puppet-master: Upgrade the puppet master ------------------------- On pergamon: :: root@pergamon:~# /usr/local/bin/deploy.sh -v .. _apply-the-configuration-on-workers: Apply the configuration on workers ---------------------------------- :: root@pergamon:~# clush -b -w @staging-loader-workers puppet agent -t .. _restart-listers: Restart the listers ------------------- On ``pergamon``, with ``clush`` :: clush -b -w @staging-loader-workers 'systemctl restart swh-worker@lister' .. _create-model-and-update--scheduler: Create the model and update the scheduler ----------------------------------------- At this stage, the listers are up to date but not yet ready to accept new tasks. The scheduler must be updated with the new lister task type. .. _upgrade-scheduler-packages: Upgrade the scheduler server packages ------------------------------------- On the scheduler server (``scheduler0.internal.staging.swh.network`` on staging) with the \*\ ``root``\ \* user: :: # Update the lister package to refresh the cli tool apt-get update apt-get install -y python3-swh.lister # check the version is the same as the workers dpkg -l python3-swh.lister .. _configure-scheduler: Configure the scheduler ----------------------- - The lister task type must be registered in the scheduler, trigger the `swh scheduler task-type register` command, on the scheduler server with ``swhscheduler`` user: :: swh scheduler --config-file /etc/softwareheritage/scheduler.yml \ task-type register Example: :: swhscheduler@scheduler0:~$ swh scheduler \ --config-file /etc/softwareheritage/scheduler.yml \ task-type register ... INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler .. _register-new-task: Register a new task ------------------- The listers and the scheduler are now ready to work together. Use the `swh scheduler task add` command. Check ref:`swh-devel:register-task-type` for more details. .. _check-logs: Check the logs -------------- On ``pergamon`` with ``root`` (or as a sudo user): :: clush -b -w @staging-loader-workers 'systemctl status swh-worker@lister' It will output the status of the listers and the last lines of the lister's logs on each worker server. diff --git a/sysadm/user-management/keycloak/authentication.rst b/sysadm/user-management/keycloak/authentication.rst index bfa937d..d693523 100644 --- a/sysadm/user-management/keycloak/authentication.rst +++ b/sysadm/user-management/keycloak/authentication.rst @@ -1,66 +1,66 @@ .. _keycloak: Authentication ============== .. admonition:: Intended audience :class: important Staff members .. contents:: :depth: 3 .. Software Heritage uses `Keycloak `__, an open source identity and access management solution, to identify and authenticate users on its services (for instance the -`archive's Web API `_ +:swh_web:`archive's Web API ` and the :ref:`deposit server `). Keycloak implements the `OpenID Connect `__ specification, a simple identity layer on top of the OAuth 2.0 protocol. It allows to get single sign-on (SSO) on various services. The base URL to interact with that authentication service is https://auth.softwareheritage.org/auth/. Introduction ------------ Keycloak defines three important concepts to know about: Realm It manages a set of users, credentials, roles, and groups. A user belongs to and logs into a realm. Realms are isolated from one another and can only manage and authenticate the users that they control. Client Entities that can request Keycloak to authenticate a user. Most often, clients are applications and services that want to use Keycloak to secure themselves and provide a single sign-on solution. Clients can also be entities that just want to request identity information or an access token so that they can securely invoke other services on the network that are secured by Keycloak. Role It identifies a type or category of users. Applications (e.g. webapp, deposit) often assign access and permissions to specific roles rather than individual users as dealing with users can be too fine grained and hard to manage. There is a global namespace for roles and each client also has its own dedicated namespace where roles can be defined. .. _software_heritage_realms: Software Heritage Realms ------------------------ Two realms are available for Software Heritage: - `SoftwareHeritageStaging `__, for testing purposes - `SoftwareHeritage `__, for production use The links above target the Admin console of each realm from which everything can be configured.