The current version of prometheus statsd exporter we're using in prod is "fairly" ancient. Among other things, it doesn't support (and silently drops!) metrics with inconsistent label sets.
Unfortunately, the changes in the core/git loader have introduced such metrics. So we should upgrade the version of our local statsd-exporter package to support them.
I've started working on this, but this needs a backport of a newer prometheus client library, so it's a bit involved.
My debian package attempts went nowhere, it seems that one of the 3rd level dependencies needs a newer golang than what's available in buster-backports, so we'll get rid of the (homegrown anyway) debian package and go the upstream binaries route.
Puppet ran manually on staging workers now, and the new statsd exporter has been deployed. The metrics properly show up in prometheus.