Centralize the postgresql databases used for admin tools into a single dedicated host
(on which we would be able to do more proper backups and monitoring).
This is also the occasion to use the latest postgresql versions (each have their own and
inconsistent versions).
Impacted services:
- hedgedoc (host: bardo, db: postgres-12)
- netbox (host: bojimans, db: postgres-11)
- grafana (host: pergamon, db: postgres-11)
- sentry (host: riverside, db: postgres-12)
- keycloak (host: kelvingrove, db: postgres-12)
This is the size of the current databases:
| Database | Size |
| ---------- | ------- |
| hedgedoc | 42MB |
| netbox | 22MB |
| grafana | 18MB |
| sentry | 99GB |
| keycloak | 15MB |
| icinga2 | 1291MB |
| icingaweb2 | 8221 kB |
The sentry database will force us to reserve a large amount of disk for the new server
Plan:
- [x] Leave services' configuration untouched to use local db
- [x] D6907, D6906: First create host database machine (vm).
- [x] T3833#76853: Create zfs data mount point (for the dbs' data)
- [x] D6928: Declare a dedicated puppet profile, this lists all
required dbs to create (using profile::postgresql::server).
- [x] D6906: terraform (/vagrant) to boostrap (this applies puppet so the dbs get created)
- firewall: Open flux from vlan 440 (bojimans, kelvingrove, riverside, pergamon) to vlan
442, port 5432
- for each service in {netbox, hedgedoc, grafana, sentry, keycloak}:
- Stop the service (we don't have incremental dump so stop the service first)
- Export and mount back data dump from old db to the new one
- Adapt configuration to switch to the new db
- puppet apply to restart service (which now uses the new db)
- Ensure service is still ok
Annex actions (outside of the main scope of the task, like T3817):
- Move services {netbox, sentry, keycloak} in the admin vlan (442) and behind the
reverse proxy
Related to D6871#178665