Page MenuHomeSoftware Heritage

JournalClient: add a stop_at_eof boolean to read the log only once
ClosedPublic

Authored by seirl on Feb 26 2020, 12:29 PM.

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

swh/journal/client.py
179–182

Here I could have used an exception instead of returning a tuple. Any thoughts?

swh/journal/tests/test_client.py
54

Most of this is copy pasted from the function above, but I'm not sure if it's worth to factor, and *how* we should factor it. I can write a fixture that yields a kafka_prefix maybe?

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/journal/client.py
213–214

This code is unsound. Your client may reach EOF on some partitions, then get rebalanced on other partitions, reach EOF on some other partitions; but it will count the older partitions on the left side.
You should try to do a set equality instead.

This revision now requires changes to proceed.Feb 26 2020, 1:48 PM
This revision is now accepted and ready to land.Feb 26 2020, 4:17 PM
This revision was landed with ongoing or failed builds.Feb 26 2020, 4:26 PM
This revision was automatically updated to reflect the committed changes.