Page MenuHomeSoftware Heritage

Use better kafka producer semantics in the journal writers
ClosedPublic

Authored by olasd on Tue, Mar 10, 7:41 PM.

Details

Summary

What we really want is for the broker to acknowledge all messages before we go
on to the next step. That's accomplished by flushing the producer rather than
enabling idempotence (which has other side-effects, such as only-once delivery,
which we don't really care about as all our consumers are, in effect,
idempotent).

Setting acks to all means that the broker acknowledges that all in-sync replicas
have persisted the message, which is a stronger guarantee than what we had
before.

Depends on D2808

Test Plan

This seems to have alleviated our broker memory consumption issues in
production.

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Tue, Mar 10, 7:41 PM
olasd updated this revision to Diff 9977.Tue, Mar 10, 7:44 PM

Rebase

ardumont accepted this revision.Wed, Mar 11, 8:01 AM
This revision is now accepted and ready to land.Wed, Mar 11, 8:01 AM