Maniphest T2128

Monitor journal consumer lag
Closed, MigratedEdits Locked
Actions

Description

Add consumer offset probe or dashboardize existing one.

Map known consumer ids to "high level" features (standardize? how?)

Related Objects

Mentioned In: T1278: swh-journal: the monitoring tool question!
Mentioned Here: T3502: Date overflow error in scheduler journal client

Event Timeline

olasd created this task.Dec 2 2019, 2:28 PM

vlorentz triaged this task as High priority.Dec 3 2019, 3:17 PM

vlorentz added a project: Metrics/monitoring.Dec 3 2019, 3:23 PM

vlorentz lowered the priority of this task from High to Normal.Dec 3 2019, 5:50 PM

A quick test shows that https://github.com/braedon/prometheus-kafka-consumer-group-exporter does a decent job.

olasd mentioned this in T1278: swh-journal: the monitoring tool question!.Dec 7 2019, 6:29 PM

Packaged and deployed the consumer group exporter on getty for both kafka clusters.

Sample query:

http://pergamon.internal.softwareheritage.org:9090/graph?g0.range_input=15m&g0.expr=sum%20without(partition)%20(kafka_topic_highwater%20-%20%20ignoring%20(group)%20group_right%20kafka_consumer_group_offset)&g0.tab=0

This would have caught T3502 earlier too.

olasd merged a task: T1278: swh-journal: the monitoring tool question!.Aug 26 2021, 12:30 PM

olasd added subscribers: ardumont, rdicosmo, zack.

This task has been migrated to GitLab.