Page MenuHomeSoftware Heritage

Rewrite the munin stats export for the website to use prometheus
Closed, ResolvedPublic

Description

Once the swh-storage metric has been integrated in prometheus, the stats export should use it.

Event Timeline

olasd triaged this task as Normal priority.Dec 11 2018, 1:17 PM
olasd created this task.
ftigeot changed the task status from Open to Work in Progress.Jul 18 2019, 4:10 PM
ftigeot added a subscriber: ftigeot.

Prometheus data has been exported to a json file similar to the format produced by the Muni/RRD based toolchain.
Results are visible on https://www-dev.softwareheritage.org/archive/
(vs https://www.softwareheritage.org/archive/ for original graphs)

The new toolchain works but we can see a few problems:

  • The graph stops at the first hole in the Prometheus dataset (21 march 2019)
  • We are missing historical values from October 2015 to December 2018 (not present in Prometheus)
  • Even for the first serie of Prometheus data, the form of the graph is too flat

For reference, a "Source files" graph produced by Prometheus itself can be visualized here:
http://pergamon.internal.softwareheritage.org:9090/graph?g0.range_input=1y&g0.end_input=2019-07-16%2015%3A13&g0.stacked=0&g0.expr=sql_swh_archive_object_count%7Bobject_type%3D%22content%22%7D&g0.tab=0

For the "March 2019 problem", the json output generated from the Prometheus API itself misses the more recent data points.

ftigeot claimed this task.Jul 23 2019, 5:38 PM

Depending on Prometheus for all data is not a hard requirement.

If it turns out to be impossible to inject historical data in Prometheus, the json creation script itself is authorized to provide the missing timecounter values.

ftigeot closed this task as Resolved.Aug 8 2019, 5:26 PM