Page MenuHomeSoftware Heritage

Graph object count per object storage backend
Closed, MigratedEdits Locked

Description

We need an estimate of the current object count in all our object storage backends, and ideally to have some trend graphs on this data.

Event Timeline

olasd triaged this task as High priority.Mar 26 2019, 4:31 PM
olasd created this task.
olasd changed the task status from Open to Work in Progress.Mar 26 2019, 6:52 PM

https://grafana.softwareheritage.org/d/jScG7g6mk/objstorage-object-counts shows the data that we're currently able to collect.

The Grafana Azure metrics data source is not great (30 days history only, data full of random holes, no aggregation capabilities, ...) so I'm considering using the Prometheus Azure metrics exporter (https://github.com/RobustPerception/azure_metrics_exporter) to properly pull the data into prometheus. The current status of that is...

azure.go         | 24 ++++++++++++------------
config/config.go |  2 ++
main.go          | 26 ++++++++++++++------------
utils.go         |  2 +-
4 files changed, 29 insertions(+), 25 deletions(-)