HomeSoftware Heritage

Commit kafka messages which offset has reach the high limit

Description

Commit kafka messages which offset has reach the high limit

this is necessary to ensure these messages are committed in kafka,
otherwise, since the (considered) empty partition is unsubscribed from,
it never gets committed in JournalClient.handle_messages() (since this
later only commit assigned partitions).

Ensure offset are committed only after worker_fn is executed without
error.

This requires to overload the JournalClient.handle_messages() method in
JournalClientOffsetRanges to make sure "pending" messages are
committed after the proper execution of worker_fn.

Doing so, we can both unsubscribe from "eof" partitions on the fly (with
"eof" meaning when the partition has been consumed up to the high
watermark offset at the beginning of the export), and commit ALL offsets
that need to be, but only after proper execution of the worker_fn
callback.

This should guarantee proper and consistent behavior (famous last
word...).

Details

Provenance
douarddaAuthored on Sep 9 2021, 2:24 PM
douarddaPushed on Sep 14 2021, 11:22 AM
Differential Revision
D6247: Commit kafka messages which offset has reach the high limit
Parents
rDDATASETa3c1f39013ba: Add a JournalClientOffsetRanges.unsubscribe() method
Branches
Unknown
Tags
Unknown
Build Status
Buildable 23572
Build 36772: test-and-buildJenkins console · Jenkins