Page MenuHomeSoftware Heritage

Move grafana on a dedicated server behind the admin RP
ClosedPublic

Authored by vsellier on Dec 22 2021, 7:39 PM.

Details

Summary

Move grafana on a dedicated server behind the admin RP

  • use the swh postgresql profile to manage the datase
  • install grafana, the plugin and the generated dashboard with puppet
  • use the rp1 reverse proxy to expose the service

Related to T3817

Test Plan

This diff doesn't not impact the current grafana instance.

diff origin/production/pergamon.softwareheritage.org current/pergamon.softwareheritage.org
*******************************************
  File[/etc/bind/keys/local-update] =>
   parameters =>
     content =>
      @@ -2,4 +2,4 @@
       key local-update {
       	algorithm hmac-sha256;
      -	secret "<redacted>";
      +	secret "<redacted>";
       };
*******************************************
  File[/etc/bind/rndc.key] =>
   parameters =>
     content =>
      @@ -2,4 +2,4 @@
       key rndc-key {
       	algorithm hmac-md5;
      -	secret "<redacted>";
      +	secret "<redacted>";
       };
*******************************************
*** End octocatalog-diff on pergamon.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier created this revision.
vsellier retitled this revision from Move grafana on a dedicated server behind the admin RP **WIP** TODO: - add auto generated dashboard - install the grafana-piechart-panel plugin to Move grafana on a dedicated server behind the admin RP.Dec 22 2021, 7:41 PM
vsellier edited the summary of this revision. (Show Details)

add the grafana-piechart-panel plugin installation

install the auto-generated dashboards with puppet

allow the database monitoring

Why the duplication of profile::grafana into profile::grafana::backend?

Do we really need the "swh postgresql" stuff for this very basic database? I find it much harder to follow than just having the existing two postgresql resources in profile::grafana.

Alternatively we should consider centralizing the postgresql databases used for admin stuff (we already have one for hedgedoc, one for netbox, and potentially the one for sentry) into a single dedicated host (on which we would be able to do more proper backups and monitoring).

data/hostname/grafana0.internal.admin.swh.network.yaml
14 ↗(On Diff #24918)

"storage"?

20–21 ↗(On Diff #24918)

No such snippet exists.

ardumont added a subscriber: ardumont.

Alternatively we should consider centralizing the postgresql databases used for admin stuff (we already have one for hedgedoc, one for netbox, and potentially the one for sentry) into a single dedicated host (on which we would be able to do more proper backups and monitoring).

I've opened T3833 for this good remark.

  • fix database name (not directly used by the configuration)
  • fix the prometheus snippets configuration

lgtm

one question inline about the pg version.

Also, in your vagrant tryouts, did you try to mount back the dump out of pergamon in the newly created grafana db?

data/hostname/grafana0.internal.admin.swh.network.yaml
2 ↗(On Diff #24969)

Since we are creating that new one, can't we directly go to 14 now (for one less db to upgrade later)?

This revision is now accepted and ready to land.Thu, Jan 6, 3:28 PM

thanks for the validation, I have some pending changes in progress and to reply to the olasd's remarks so I change the status to planned changes

Regarding the backup restoration in vagrant, yes I have successfully restored it locally, everything was good (except the data of the graphes because grafana request my empty local prometheus)

  • Fix the database password resolution after the database name update
  • Restore the profile::grafana::objects call to manage the orgs and database declarations It's not ideal as it introduces a dependency with the reverse proxy
This revision is now accepted and ready to land.Fri, Jan 7, 9:47 AM
  • Use the postgresql module to configure the database Indeed, it's simpler
  • fix a wrong hostname declaration for vagrant
  • fix grafana plugin installation and configuration calls

Add more documentation on the backend profile

Why the duplication of profile::grafana into profile::grafana::backend?

The goal was to let the initial profile untouched during the migration and to clearly identify the public reverse_proxy is not installed by this part

Do we really need the "swh postgresql" stuff for this very basic database? I find it much harder to follow than just having the existing two postgresql resources in profile::grafana.

Nope you are right, it was clearly overkill

  • Rebase
  • Use the centralized database instead of a local postgresql
  • Move the prometheus sql exporter on dali
This revision is now accepted and ready to land.Fri, Jan 14, 10:26 AM