Page MenuHomeSoftware Heritage

Make the default value for the kafka writer's message.max.bytes 100MB
ClosedPublic

Authored by olasd on Apr 9 2020, 2:13 PM.

Details

Summary

This matches the current configuration everywhere, and the future configuration
of brokers.

Depends on D2992.

Test Plan

This is already the production configuration on existing writers

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This revision is now accepted and ready to land.Apr 9 2020, 2:25 PM

Build is green

Patch application report for D2993 (id=10632)

Could not rebase; Attempt merge onto 893d285adf...

Updating 893d285..e9d970f
Fast-forward
 swh/journal/serializers.py             |  6 ++++--
 swh/journal/tests/test_kafka_writer.py |  2 --
 swh/journal/writer/kafka.py            | 21 ++++++++++++++-------
 3 files changed, 18 insertions(+), 11 deletions(-)
Changes applied before test
commit e9d970ffb1594b98578e6011da0a93627be4deca
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 15:22:37 2020 +0200

    Make the default value for the kafka writer's message.max.bytes 100MB
    
    This matches the current configuration everywhere, and the future configuration
    of brokers.

commit 7b9dc4d477b994846c7027481c96e949bbda7fd9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 17:00:42 2020 +0200

    Use Optional[x] instead of Union[x, None]

commit e1ab119cc51aec0440ffaf8431e05b2e60609763
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 16:58:51 2020 +0200

    Introduce a KeyType alias in serializers to factor out its definition

commit 08348047d28bfd699a827f6a6b0fcf5dc8625aca
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 16:50:57 2020 +0200

    Accept Sequence[ModelObject] instead of List[ModelObject]
    
    The latter is invariant (because List is a mutable type), so it doesn't allow
    passing in lists of ModelObject subtypes.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/24/ for more details.

Build is green

Patch application report for D2993 (id=10638)

Could not rebase; Attempt merge onto 893d285adf...

Updating 893d285..7a61d21
Fast-forward
 swh/journal/serializers.py             |  6 ++++--
 swh/journal/tests/test_kafka_writer.py |  2 --
 swh/journal/writer/kafka.py            | 21 ++++++++++++++-------
 3 files changed, 18 insertions(+), 11 deletions(-)
Changes applied before test
commit 7a61d21811f61b5f3579007487d46dfc532907b4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 15:22:37 2020 +0200

    Make the default value for the kafka writer's message.max.bytes 100MB
    
    This matches the current configuration everywhere, and the future configuration
    of brokers.

commit 0188902962eb7cd91d471fdd3984b13ab235339a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 17:00:42 2020 +0200

    Use Optional[x] instead of Union[x, None]

commit bd966490570c3d07df2e3adbb81b83ef3db2aed7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 16:58:51 2020 +0200

    Introduce a KeyType alias in serializers to factor out its definition

commit eb4f3626e8f3f6e0cffbc610fdd47e06ddbdac22
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 16:50:57 2020 +0200

    write_additions: Accept Iterable[ModelObject] instead of List[ModelObject]
    
    This allows us to pass objects typed as lists of ModelObject subtypes.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/29/ for more details.