Page MenuHomeSoftware Heritage

Install grafana on its own server
Closed, MigratedEdits Locked

Description

Install grafana on its own server.

  • in the admin vlan
  • behind rp1
  • bullseye

Event Timeline

vsellier changed the task status from Open to Work in Progress.Dec 22 2021, 9:42 AM
vsellier triaged this task as Low priority.
vsellier created this task.

The diffs are ready to be reviewed.
The migration will be performed at the beginning of January.

Actions to perform:

  • land the D6871 diff when validated
  • provision the new vm with terraform
  • apply puppet on rp1
  • Create the following rules on the firewall
    • VLAN442:
      • icinga server -> grafana server:postgresql (to validate if it's really needed)
    • VLAN440:
      • grafana server -> prometheus server:prometheus port
      • grafana server -> elasticsearch nodes:elasticsearch port
  • dump the grafana database on pergamon
sudo -i -u postgres pg_dump -c -C grafana > grafana.sql

(-c -C to recreate the database on the new vm as it will be already created by puppet)

  • import the database on grafana0
cat grafana.sql | sudo -i -u postgres pgsql
  • check evrything is ok by overriding adding the following line in your /etc/hosts
192.168.50.20 grafana.softwareheritage.org`
  • if everything if ok, change the public dns entry on the softwareheritage.org domain to
grafana CNAME 1800 swh-rproxy3.inria.fr.

If everything is ok, prepare a diff to cleanup the prometheus node / manual cleanup

  • remove the grafana vhosts from apache
  • remove the following packages
    • grafana
    • swh-grafanalib-dashboards
  • drop the grafana database from postgresql
  • remove the following directories:
    • /var/lib/grafana
    • /etc/grafana

We will wait for the availability of the dedicated admin database server (T3833) before moving grafana. It will avoid to move the data 2 times and will upgrade the database to the last postgresql version.

ardumont raised the priority of this task from Low to Normal.Jan 14 2022, 11:15 AM
ardumont moved this task from Weekly backlog to in-progress on the System administration board.
  • grafana0 server created
  • configuration applied on the reverse proxy
  • network configuration checks:
    • grafana0 -> pergamon:9090 (prometheus): Allowed for all in a floating rule
    • grafana0 -> esnode[1-3]:9200 (elasticsearch queries):
  • public DNS updated to use rp1 (swh-rproxy3.inria.fr) as the public entry point
  • pergamon apache configuration temporary updated to point to the new grafana0 backend server ( and add a temporary rule on the firewall to allow it)
  • pergamon cleanup done with D6952.

Apache vhosts were automatically removed

The database removal will be handled in T3849