Page MenuHomeSoftware Heritage

Add extra configuration to kafka consumer from journal.client
AbandonedPublic

Authored by anlambert on Apr 5 2019, 6:00 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Summary

While playing with kafka components in the docker-compose environment, I often stumbled
across the issue described in T1575. Setting the max_poll_records configuration entry
to the maximum number of messages to process in a batch fixes that issue.

I also observed that applications can hang while iterating on a consumer for messages.
Indeed, default kafka behavior is to wait undefinitely for new messages while iterating.
This can result in some messages being consumed but not processed (typically when the
maximum number of messages to process is not reached) until new messages to consume
are available. Setting a value for the consumer_timeout_ms configuration entry
ensures that application will not hang when such a situation happens.

Related T1575

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
kafka-config-improvement
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5268
Build 7133: tox-on-jenkinsJenkins
Build 7132: arc lint + arc unit

Event Timeline

Tests for the application hanging scenario should be added to better detail the observed issue, so planning changes to that diff for next week.

ardumont requested changes to this revision.EditedApr 9 2019, 1:57 PM
ardumont added a subscriber: ardumont.

As the publisher is going away, this needs to be rebased on top of D1372's content when it lands (to integrate the journal client part).

Rebase to master, still some tests to journal.client to add

anlambert retitled this revision from Add extra configuration to kafka consumers to Add extra configuration to kafka consumer from journal.client.