Page MenuHomeSoftware Heritage

Make the journal writer reliable
Closed, MigratedEdits Locked

Description

When initially developing the journal writer based on confluent-kafka-python, the semantics of delivery callbacks, KafkaErrors and whether they are fatal was misunderstood.

To get reliable delivery of messages, we need to make sure that we listen to delivery callbacks, and bubble up any error associated with messages (rather than generic errors). The errors for message non-persistence are not marked as fatal (...).