Page MenuHomeSoftware Heritage

Most indexers are consuming journal topics slower than messages are produced
Closed, MigratedEdits Locked

Description

Grafana dashboards show that most indexer consumers are working, but they are lagging behind, and the gap is increasing.

The plot of lag derivative shows a progressive slowdown, so it's probably not due to a specific configuration change.

I do not know what is causing this, though. Two possible suspects:

  1. rdkafka frequently disconnecting from the brokers (or generally, having connection isues): https://sentry.softwareheritage.org/share/issue/76ed328b2ae6465face2ea4bb5f32187/
  2. slow storage and/or objstorage (which would make sense, as the extrinsic-metadata indexer is super-fast, and is also the only one not to use the storage and objstorage)
  3. simply not having enough workers

Event Timeline

vlorentz triaged this task as Normal priority.Oct 7 2022, 12:08 PM
vlorentz created this task.
vlorentz updated the task description. (Show Details)