HomeSoftware Heritage

Handle BufferErrors in writer.kafka.Producer

Description

Handle BufferErrors in writer.kafka.Producer

When the local rdkafka producer queue is full, the producer would crash
with a BufferError. While this condition is rare, it can happen when
running a backfill operation.

When such a condition arises, retry sending the message a few times with
an increasing backoff. When giving up, use the common KafkaDeliveryError
mechanism rather than the plain BufferError.

Details

Provenance
olasdAuthored on Feb 25 2021, 12:12 PM
olasdPushed on Feb 25 2021, 6:48 PM
Parents
rDJNLed9d6827af6f: Remove Makefile.local
Branches
Unknown
Tags
Unknown
Build Status
Buildable 19502
Build 30253: test-and-buildJenkins console · Jenkins