Page MenuHomeSoftware Heritage

Add loader-related metrics to swh-loader-core
Closed, MigratedEdits Locked

Description

Add some metrics to swh-loader-core

  • fetch duration (histogram)
  • number of objects of each kind added to the archive (counters)

Event Timeline

olasd triaged this task as High priority.Jan 8 2019, 11:45 AM
olasd created this task.
olasd removed olasd as the assignee of this task.Sep 22 2020, 6:10 PM
olasd updated the task description. (Show Details)
olasd updated the task description. (Show Details)

I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.

I'm not sure what kind of granularity we really want for the loader-level object counters (do we want per-loader breakdowns of objects added? per-visit?)

Once that's specified this is probably a candidate for an easy hack (as most of the data is there, it just needs to be collected / sent to statsd).

vlorentz claimed this task.
vlorentz added a subscriber: vlorentz.

Closing this, as these metrics are now visible on https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics

There is still room for finer metrics though, such as T4185