Differential D5144

Handle BufferErrors in writer.kafka.Producer
ClosedPublic
Actions

Authored by olasd on Feb 25 2021, 3:11 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3000: Investigate scheduler journal client discrepancies

Commits

rDJNL458a405afb01: test_kafka_writer: remove useless fixtures

Summary

When the local rdkafka producer queue is full, the producer would crash
with a BufferError. While this condition is rare, it can happen when
running a backfill operation.

When such a condition arises, retry sending the message a few times with
an increasing backoff. When giving up, use the common KafkaDeliveryError
mechanism rather than the plain BufferError.

Related to T3000

Test Plan

new tests added for retry and DeliveryError

Diff Detail

Repository

rDJNL Journal infrastructure

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Feb 25 2021, 3:11 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 25 2021, 3:11 PM

Build is green

Patch application report for D5144 (id=18391)

Rebasing onto ed9d6827af...

First, rewinding head to replay your work on top of it...
Applying: Handle BufferErrors in writer.kafka.Producer

Changes applied before test

commit 0bd690710311086f6ee2e1ba193604ef03feb654
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Feb 25 12:12:35 2021 +0100

    Handle BufferErrors in writer.kafka.Producer
    
    When the local rdkafka producer queue is full, the producer would crash
    with a BufferError. While this condition is rare, it can happen when
    running a backfill operation.
    
    When such a condition arises, retry sending the message a few times with
    an increasing backoff. When giving up, use the common KafkaDeliveryError
    mechanism rather than the plain BufferError.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/150/ for more details.

Harbormaster completed remote builds in B19469: Diff 18391.Feb 25 2021, 3:14 PM

olasd requested review of this revision.Feb 25 2021, 3:14 PM

ardumont accepted this revision.Feb 25 2021, 3:38 PM

This revision is now accepted and ready to land.Feb 25 2021, 3:38 PM

rebase

Build is green

Patch application report for D5144 (id=18420)

Could not rebase; Attempt merge onto ed9d6827af...

Updating ed9d682..458a405
Fast-forward
 swh/journal/tests/test_kafka_writer.py | 70 +++++++++++++++++++++++++++++++---
 swh/journal/writer/kafka.py            | 32 ++++++++++++++--
 2 files changed, 92 insertions(+), 10 deletions(-)

Changes applied before test

commit 458a405afb01bee19c2f8ea6bb3ddc997c838b0c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Feb 25 15:13:48 2021 +0100

    test_kafka_writer: remove useless fixtures

commit 6001a28f7d81356bed3b528bd1c2f8db101b031c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Feb 25 12:12:35 2021 +0100

    Handle BufferErrors in writer.kafka.Producer
    
    When the local rdkafka producer queue is full, the producer would crash
    with a BufferError. While this condition is rare, it can happen when
    running a backfill operation.
    
    When such a condition arises, retry sending the message a few times with
    an increasing backoff. When giving up, use the common KafkaDeliveryError
    mechanism rather than the plain BufferError.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/151/ for more details.

Harbormaster completed remote builds in B19498: Diff 18420.Feb 25 2021, 6:46 PM

Closed by commit rDJNL458a405afb01: test_kafka_writer: remove useless fixtures (authored by olasd). · Explain WhyFeb 25 2021, 6:48 PM

This revision was automatically updated to reflect the committed changes.

olasd added a commit: rDJNL458a405afb01: test_kafka_writer: remove useless fixtures.