Page MenuHomeSoftware Heritage

Configure db1 as new staging db server
Closed, ResolvedPublic

Description

We want to aligned the staging db as closed as possible than the production server

The following datasets will be configured configured :

  • (already created) Dataset /data/postgres-main-12 mount on /srv/softwareheritage/postgres/12/main (postgresql:5433)
    • softwareheritage
  • Dataset /data/postgres-secondary-12 mount on /srv/softwareheritage/postgres/12/secondary (postgresql:5434)
    • swh-vault
    • swh-lister
    • softwareheritage-scheduler
    • softwareheritage-deposit
  • Dataset data/postgres-indexer-12 mounted on /srv/softwareheritage/postgres/12/indexer (postgresql:5435) [1]
    • softwareheritage-indexer

[1] On production the indexer is using postgresql 11 at the moment

  • Puppet
    • change the db0.internal.staging.swh.network node to use a wildcard selector
    • configure the new database node as the db host for the swh services

Revisions and Commits

rDENV Development environment
D4394
D4394
D4394
D4394
D4394
D4394
D4377
rDCIDX Metadata indexer
D4378
rDVAU Software Heritage Vault
D4379
rSPSITE puppet-swh-site
D4391
D4372
D4375
D4370
D4370
D4370
D4370
rDCORE Foundations and core functionalities
D4374

Event Timeline

vsellier changed the task status from Open to Work in Progress.Oct 26 2020, 4:18 PM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier created this object with edit policy "System administrators (Project)".
  • Create the postgresql:5434 dataset
zfs create data/postgres-secondary-12 -o mountpoint=/srv/softwareheritage/postgres/12/secondary
  • Create the postgresql:5435 dataset
zfs create data/postgres-indexer-12 -o mountpoint=/srv/softwareheritage/postgres/12/indexer
  • check datasets :
root@db1:~# zfs list -t all
NAME                         USED  AVAIL     REFER  MOUNTPOINT
data                        1.20M  26.4T       96K  /data
data/postgres-indexer-12      96K  26.4T       96K  /srv/softwareheritage/postgres/12/indexer
data/postgres-main-12         96K  26.4T       96K  /srv/softwareheritage/postgres/12/main
data/postgres-misc           104K  26.4T      104K  /srv/softwareheritage/postgres
data/postgres-secondary-12    96K  26.4T       96K  /srv/softwareheritage/postgres/12/secondary

For the puppet part, the actual staging configuration needs some adaptations as the configuration install postgresql on version 11 and 13. Another point is the different clusters are not managed by puppet but it's the same for the production.

At least, the configuration needs to be adapted to install only the version 12 to have the same version as in the production environment (except for the indexer still using the version 11).
We will dig with @ardumont if it's also possible to manage the 3 psql clusters via puppet in a reasonable amount of time

For the puppet part, the actual staging configuration needs some adaptations as the configuration install postgresql on version 11 and 13

Fixed by D4359.

The puppetlabs-postgresql module doesn't allow to manage several postgresql clusters. We have made the tradeoff to use only one cluster on db1 at the beginning to be able to deploy db1 via puppet as it's the priority. The module will be extended or replaced by something else later.

Road so far, after the puppet run which creates the dbs with guest, postgres
and swh-admin users.

As tryouts, from staging-db1 node (and a bit on the actual db1, which
we'll finish during the day):

$ cat init-db.sh
#!/usr/bin/env bash

set -x

export PGUSER=postgres
# export PGHOST=$(ip a | grep eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1)
export PGHOST=db1.internal.staging.swh.network
export PGPORT=5432
# export PGPASSWORD=swh::deploy::db::postgres::password
export PGPASSWORD=<insert-postgres-pass-here>

MODULE=${1}

if [ "${MODULE}" == "storage" ]; then
   DBNAME="swh"
else
   DBNAME="swh-${MODULE}"
fi

# Connect as super-admin to init the extensions
swh db init-admin --db-name $DBNAME $MODULE   # should be changed to init-admin

# Connect as owner of the db to init the schema
export PGUSER=$DBNAME
export PGPASSWORD=${2}
swh db init --db-name $DBNAME $MODULE

# Connect as owner to grant read access to schema to guest user
psql -c "grant select on all tables in schema public to guest;"

The actual initialization calls to bootstrap the different schema below (the
second parameter is the password, for vagrant node, coming from the censored
private data repository, for the actual production, this will be adapted
accordingly):

$ ./init-db.sh scheduler swh::deploy::scheduler::db::password
DONE database for scheduler initialized at version 17
$ ./init-db.sh indexer swh::deploy::indexer::storage::db::password
DONE database for indexer initialized at version 132
$ ./init-db.sh vault swh::deploy::vault::db::password
DONE database for vault initialized at version 1
$ ./init-db.sh storage swh::deploy::storage::db::password
DONE database for storage initialized (flavor default) at version 163

Note:

  • swh db create is used for now, the code on the node is patched to comment out the sql create database call to prevent breaking (D4374).
  • the swh-admin superuser is used to allow privilege installation setup (D4375)

Other dbs have their own initialization way:

Staging nodes are now using storage1 (and db1).
So db0 and storage0 can be decomissionned (and thus orsay \o/).

We triggered some loading (swh loader run git..., some are still running btw, linux, nixpkgs, guix, ...).
Icinga checks have been fixed a bit... (firewall rule update, ...)

We need to shake some more other services (save code now, cooking task, trigger listings, ... )

Following the diff D4391, the zfs dsatasets were reconfigured tobe mounted on the /srv/softwareheritage/postgres/* :

systemctl stop postgresql@12-main
zfs set mountpoint=none data/postgres-indexer-12
zfs set mountpoint=none data/postgres-secondary-12
zfs set mountpoint=none data/postgres-main-12
zfs set mountpoint=none data/postgres-misc

zfs set mountpoint=/srv/softwareheritage/postgres data/postgres-misc
zfs set mountpoint=/srv/softwareheritage/postgres/12/indexer data/postgres-indexer-12
zfs set mountpoint=/srv/softwareheritage/postgres/12/secondary data/postgres-secondary-12
zfs set mountpoint=/srv/softwareheritage/postgres/12/main data/postgres-main-12

systemctl start postgresql@12-main

It must be done in this order because there is a hierarchical mount between postgres/ and postgres/*

We have performed with @ardumont several tests on the webapp, the vault, the deposit, the loaders and the listers and it seems everything is working well.

The remaining monitoring alerts are due to repositories not existing anymore and not easily importable. The monitored urls will be updated to something more easy to ingest in a separate tasks.

this one can be closed.