Add some metrics to swh-loader-core
- fetch duration (histogram)
- number of objects of each kind added to the archive (counters)
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Open | None | T1408 More/better Metrics | ||
| Resolved | olasd | T1535 Deploy prometheus-statsd-exporter to gather per-worker metrics | ||
| Resolved | vlorentz | T1461 Add loader-related metrics to swh-loader-core |
Event Timeline
Comment Actions
I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.
I'm not sure what kind of granularity we really want for the loader-level object counters (do we want per-loader breakdowns of objects added? per-visit?)
Once that's specified this is probably a candidate for an easy hack (as most of the data is there, it just needs to be collected / sent to statsd).
Comment Actions
Closing this, as these metrics are now visible on https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics
There is still room for finer metrics though, such as T4185