Page MenuHomeSoftware Heritage

Deploy a more recent version of prometheus-statsd-exporter on all nodes
Closed, MigratedEdits Locked


The current version of prometheus statsd exporter we're using in prod is "fairly" ancient. Among other things, it doesn't support (and silently drops!) metrics with inconsistent label sets.

Unfortunately, the changes in the core/git loader have introduced such metrics. So we should upgrade the version of our local statsd-exporter package to support them.

Event Timeline

olasd triaged this task as High priority.May 6 2022, 4:55 PM
olasd created this task.
olasd changed the task status from Open to Work in Progress.May 6 2022, 5:00 PM
olasd claimed this task.

I've started working on this, but this needs a backport of a newer prometheus client library, so it's a bit involved.

My debian package attempts went nowhere, it seems that one of the 3rd level dependencies needs a newer golang than what's available in buster-backports, so we'll get rid of the (homegrown anyway) debian package and go the upstream binaries route.

olasd renamed this task from Package more recent version of prometheus-statsd-exporter to Deploy a more recent version of prometheus-statsd-exporter on all nodes.May 12 2022, 11:35 AM

Puppet ran manually on staging workers now, and the new statsd exporter has been deployed. The metrics properly show up in prometheus.

olasd moved this task from deployed/landed/monitoring to done on the System administration board.

That's deployed on all nodes and validated as working now.