Changeset View
Changeset View
Standalone View
Standalone View
docs/sys-info.rst
.. _swh-deposit-deployment: | .. _swh-deposit-deployment: | ||||
Deployment of the swh-deposit | Deployment | ||||
============================= | ========== | ||||
The deposit is architectured around 3 parts: | |||||
- server: a django application exposing an xml api, discussing with a postgresql | |||||
backend (and optionally a keycloak instance) | |||||
- worker(s): 1 worker service dedicated to check the deposit archive and metadata are | |||||
correct (the checker), another worker service dedicated to actually ingest the | |||||
deposit into the swh archive. | |||||
- client: a python sčript `swh deposit` command line interface. | |||||
All those are packaged in 3 separated debian packages, created and uploaded to the swh | |||||
debian repository. The deposit server and workers configuration are managed by puppet | |||||
(cf. puppet-environment/swh-site, puppet-environment/swh-role, | |||||
puppet-environment/swh-profile) | |||||
In the following document, we will focus on the server actions that may be needed once | |||||
the server is installed or upgraded. | |||||
The debian package is created and uploaded to the swh debian repository. Once the | |||||
package is installed, some further actions may be required in regards to the database | |||||
backend. | |||||
Prepare the database setup (existence, connection, etc...). | Prepare the database setup (existence, connection, etc...). | ||||
----------------------------------------------------------- | ----------------------------------------------------------- | ||||
This is defined through the packaged module ``swh.deposit.settings.production`` and the | This is defined through the packaged module ``swh.deposit.settings.production`` and the | ||||
expected **/etc/softwareheritage/deposit/server.yml** configuration file. | expected **/etc/softwareheritage/deposit/server.yml** configuration file. | ||||
The expected configuration files are deployed through our puppet manifest (cf. | |||||
puppet-environment/swh-site, puppet-environment/swh-role, | |||||
puppet-environment/swh-profile) | |||||
Environment (production/staging) | Environment (production/staging) | ||||
-------------------------------- | -------------------------------- | ||||
`SWH_CONFIG_FILENAME` must be defined and target the deposit server configuration file. | `SWH_CONFIG_FILENAME` must be defined and target the deposit server configuration file. | ||||
So either 1. prefix the following commands or 2. export the environment variable in your | So either 1. prefix the following commands or 2. export the environment variable in your | ||||
shell session. For the remaining part of the documentation, we assume 2. has been | shell session. For the remaining part of the documentation, we assume 2. has been | ||||
configured. | configured. | ||||
.. code:: shell | .. code:: shell | ||||
export SWH_CONFIG_FILENAME=/etc/softwareheritage/deposit/server.yml | export SWH_CONFIG_FILENAME=/etc/softwareheritage/deposit/server.yml | ||||
Migrate/bootstrap the db schema | |||||
------------------------------- | |||||
.. code:: shell | |||||
sudo django-admin migrate --settings=swh.deposit.settings.production | Migrate the db schema | ||||
--------------------- | |||||
Load minimum defaults data | |||||
-------------------------- | |||||
When boostraping the db schema, some default values may be needed: | The debian package may integrate some new schema modifications. To run them: | ||||
.. code:: shell | .. code:: shell | ||||
sudo django-admin loaddata \ | sudo django-admin migrate --settings=swh.deposit.settings.production | ||||
--settings=swh.deposit.settings.production deposit_data | |||||
This adds the minimal 'hal' collection | |||||
Note: swh.deposit.fixtures.deposit\_data is packaged. | |||||
Add client and collection | Add client and collection | ||||
------------------------- | ------------------------- | ||||
The deposit can now be configured to use either the 1. django basic authentication | The deposit can be configured to use either the 1. django basic authentication framework | ||||
framework or the 2. swh keycloak instance. If the server uses 2., the password is | or the 2. swh keycloak instance. If the server uses 2., the password is managed by | ||||
managed by keycloak so the option `--password`` is ignored. | keycloak so the option `--password`` is ignored. | ||||
* basic | |||||
.. code:: shell | .. code:: shell | ||||
swh deposit admin \ | swh deposit admin \ | ||||
--config-file $SWH_CONFIG_FILENAME \ | --config-file $SWH_CONFIG_FILENAME \ | ||||
--platform production \ | --platform production \ | ||||
user create \ | user create \ | ||||
--collection <collection-name> \ | --collection <collection-name> \ | ||||
--username <client-name> \ | --username <client-name> \ | ||||
--password <to-define> | --password <to-define> | ||||
This adds a user ``<client-name>`` which can access the collection | This adds a user ``<client-name>`` which can access the collection | ||||
``<collection-name>``. The password will be used for checking the authentication access | ``<collection-name>``. The password will be used for checking the authentication access | ||||
to the deposit api (if 1. is used). | to the deposit api (if 1. is used). | ||||
Note: | Note: | ||||
- If the collection does not exist, it is created alongside | - If the collection does not exist, it is created alongside | ||||
- The password, if required, is passed as plain text but stored encrypted | - The password, if required, is passed as plain text but stored encrypted | ||||
Reschedule a deposit | Reschedule a deposit | ||||
--------------------- | --------------------- | ||||
If for some reason, the loading failed, after fixing and deploying the new deposit | |||||
loader, you can reschedule the impacted deposit through: | |||||
.. code:: shell | .. code:: shell | ||||
swh deposit admin \ | swh deposit admin \ | ||||
--config-file $SWH_CONFIG_FILENAME \ | --config-file $SWH_CONFIG_FILENAME \ | ||||
--platform production \ | --platform production \ | ||||
deposit reschedule \ | deposit reschedule \ | ||||
--deposit-id <deposit-id> | --deposit-id <deposit-id> | ||||
This will: | This will: | ||||
- check the deposit's status to something reasonable (failed or done). That means that | - check the deposit's status to something reasonable (failed or done). That means that | ||||
the checks have passed but something went wrong during the loading (failed: loading | the checks have passed but something went wrong during the loading (failed: loading | ||||
failed, done: loading ok, still for some reasons as in bugs, we need to reschedule it) | failed, done: loading ok, still for some reasons as in bugs, we need to reschedule it) | ||||
- reset the deposit's status to 'verified' (prior to any loading but after the checks | - reset the deposit's status to 'verified' (prior to any loading but after the checks | ||||
which are fine) and removes the different archives' identifiers (swh-id, ...) | which are fine) and removes the different archives' identifiers (swh-id, ...) | ||||
- trigger back the loading task through the scheduler | - trigger back the loading task through the scheduler | ||||
Integration checks | Integration checks | ||||
------------------ | ------------------ | ||||
There exists icinga checks running periodically on `staging`_ and `production`_ | There exists icinga checks running periodically on `staging`_ and `production`_ | ||||
instances. If any problem arises, expect those to notify the #swh-sysadm irc channel. | instances. If any problem arises, expect those to notify the #swh-sysadm irc channel. | ||||
.. _staging: https://icinga.softwareheritage.org/search?q=deposit#!/monitoring/service/show?host=pergamon.softwareheritage.org&service=staging%20Check%20deposit%20end-to-end | .. _staging: https://icinga.softwareheritage.org/search?q=deposit#!/monitoring/service/show?host=pergamon.softwareheritage.org&service=staging%20Check%20deposit%20end-to-end | ||||
.. _production: https://icinga.softwareheritage.org/search?q=deposit#!/monitoring/service/show?host=pergamon.softwareheritage.org&service=production%20Check%20deposit%20end-to-end | .. _production: https://icinga.softwareheritage.org/search?q=deposit#!/monitoring/service/show?host=pergamon.softwareheritage.org&service=production%20Check%20deposit%20end-to-end |