Page MenuHomeSoftware Heritage

Add support for the static consumer group feature to journal client
ClosedPublic

Authored by olasd on Mar 2 2020, 7:19 PM.

Details

Summary

The new KAFKA_GROUP_INSTANCE_ID env variable can be set on journal clients to
set the group.instance.id enable support for Kafka Static Consumer Groups.

https://github.com/edenhill/librdkafka/pull/2525
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

Combined with larger values for the session.timeout.ms and max.poll.interval.ms
settings, this setting informs the Group Coordinator broker that the consumer
group has static membership, and that the disappearance of a given member of the
consumer group should not immediately trigger a rebalance; This allows crashing
consumers to re-join the consumer group and start consuming from their assigned
partitions immediately.

This setting is implemented as an environment variable so that several consumers
in the same group can share a configuration file, and still override the
value (e.g. by setting Environment=KAFKA_GROUP_INSTANCE_ID=groupname-%i in a
systemd template unit).

Test Plan

this now has a new unit test 0:-)

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz added a subscriber: vlorentz.

nice!

swh/journal/client.py
104

This probably deserves a log message (debug or info)

This revision is now accepted and ready to land.Mar 2 2020, 7:25 PM
swh/journal/client.py
104

When debugging is enabled, consumer settings are already dumped (see line 124). I guess that should be enough?

This is nice.

this probably deserves a new unit test 0:-)

Probably not easy to write, but this also shows the whole package deserves a bit of documentation. For example with examples of decent configuration for "high throughput" clients (you mention in the diff that this can be useful

Combined with larger values for the session.timeout.ms [...]

so I guess a proper documentation may be useful.

olasd edited the summary of this revision. (Show Details)

Rebase on top of D2754. Updated default settings for session.timeout.ms and max.poll.interval.ms when using static group membership.

olasd requested review of this revision.Mar 3 2020, 1:18 PM
This revision is now accepted and ready to land.Mar 3 2020, 1:20 PM