Page MenuHomeSoftware Heritage

Add loader-related metrics to swh-loader-core
Open, HighPublic

Description

Add some metrics to swh-loader-core

  • fetch duration (histogram)
  • number of objects of each kind added to the archive (counters)

Event Timeline

olasd triaged this task as High priority.Jan 8 2019, 11:45 AM
olasd created this task.
olasd removed olasd as the assignee of this task.Sep 22 2020, 6:10 PM
olasd updated the task description. (Show Details)
olasd updated the task description. (Show Details)
olasd added a comment.Sep 22 2020, 6:13 PM

I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.

I'm not sure what kind of granularity we really want for the loader-level object counters (do we want per-loader breakdowns of objects added? per-visit?)

Once that's specified this is probably a candidate for an easy hack (as most of the data is there, it just needs to be collected / sent to statsd).