Page MenuHomeSoftware Heritage

Rewrite the munin stats export for the website to use prometheus
Closed, MigratedEdits Locked

Description

Once the swh-storage metric has been integrated in prometheus, the stats export should use it.

Event Timeline

olasd triaged this task as Normal priority.Dec 11 2018, 1:17 PM
olasd created this task.
ftigeot changed the task status from Open to Work in Progress.Jul 18 2019, 4:10 PM
ftigeot added a subscriber: ftigeot.

Prometheus data has been exported to a json file similar to the format produced by the Muni/RRD based toolchain.
Results are visible on https://www-dev.softwareheritage.org/archive/
(vs https://www.softwareheritage.org/archive/ for original graphs)

The new toolchain works but we can see a few problems:

  • The graph stops at the first hole in the Prometheus dataset (21 march 2019)
  • We are missing historical values from October 2015 to December 2018 (not present in Prometheus)
  • Even for the first serie of Prometheus data, the form of the graph is too flat

For reference, a "Source files" graph produced by Prometheus itself can be visualized here:
http://pergamon.internal.softwareheritage.org:9090/graph?g0.range_input=1y&g0.end_input=2019-07-16%2015%3A13&g0.stacked=0&g0.expr=sql_swh_archive_object_count%7Bobject_type%3D%22content%22%7D&g0.tab=0

For the "March 2019 problem", the json output generated from the Prometheus API itself misses the more recent data points.

Depending on Prometheus for all data is not a hard requirement.

If it turns out to be impossible to inject historical data in Prometheus, the json creation script itself is authorized to provide the missing timecounter values.