HomeSoftware Heritage

journalprocessor: Fix deserialize_message raising EOFError on the last message…

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

journalprocessor: Fix deserialize_message raising EOFError on the last message of each assignment

This caused JournalClientOffsetRanges to ignore the last batch of messages
in each assignment, because JournalClient.handle_messages deserializes
all messages in the batch before calling the worker function;
and raising EOFError from deserialize_message makes it exit early
(before calling the worker fn).

Additionally, it doesn't make much sense for a deserialize_message
function to raise this kind of exception.

Instead, this commit removes the explicit raise EOFError, and tells
JournalClient to stop on EOF. deserialize_message calls
handle_offset, which updates the assignment of the Kafka consumer to
be the empty set, which causes it to be EOF (since there are no more
partitions to read from).

Details

Provenance
vlorentzAuthored on Jul 28 2021, 12:26 PM
vlorentzPushed on Jul 28 2021, 3:15 PM
Differential Revision
D6038: journalprocessor: Fix deserialize_message raising EOFError on the last message of each assignment

Commit No Longer Exists

This commit no longer exists in the repository.