HomeSoftware Heritage

Add support for the static consumer group feature to journal client

Description

Add support for the static consumer group feature to journal client

The new KAFKA_GROUP_INSTANCE_ID env variable can be set on journal clients to
set the group.instance.id enable support for Kafka Static Consumer Groups.

https://github.com/edenhill/librdkafka/pull/2525
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

Combined with larger values for the session.timeout.ms and max.poll.interval.ms
settings, this setting informs the Group Coordinator broker that the consumer
group has static membership, and that the disappearance of a given member of the
consumer group should not immediately trigger a rebalance; This allows crashing
consumers to re-join the consumer group and start consuming from their assigned
partitions immediately.

This setting is implemented as an environment variable so that several consumers
in the same group can share a configuration file, and still override the
value (e.g. by setting Environment=KAFKA_GROUP_INSTANCE_ID=groupname-%i in a
systemd template unit).

When this setting is enabled, we also up the relevant values for
session.timeout.ms and max.poll.interval.ms.

Details

Provenance
olasdAuthored on Feb 28 2020, 4:19 PM
olasdPushed on Mar 3 2020, 1:26 PM
Differential Revision
D2751: Add support for the static consumer group feature to journal client
Parents
rDJNLa3cc57724e34: Upgrade kafka to 2.4.0
Branches
Unknown
Tags
Unknown
Build Status
Buildable 10869
Build 16337: test-and-buildJenkins