client: allow to specify any cfg parameter to KafkaConsumer()
ClosedPublic
Actions

Authored by douardda on Jun 19 2019, 10:40 AM.

Details

Reviewers

Group Reviewers

Commits

rDJNL693b8f407fb4: cli: add a --concurrency cli option to content-replay
rDJNL022baee25827: replay: parallelize the copy of objects from a polled set
rDJNL253d150b5d0b: client: allow to specify any cfg parameter to KafkaConsumer()

Summary

directly from the config file.

This is needed because we want to be able to configure things like
max_poll_records or any other kafka tuning parameter.

also parallelize the copy of objects from a polled set in
content-replayer using a ThreadPoolExecutor.

Diff Detail

Repository

rDJNL Journal infrastructure

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jun 19 2019, 10:40 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 19 2019, 10:40 AM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/170/ for more details.

Harbormaster completed remote builds in B6311: Diff 5328.Jun 19 2019, 10:44 AM

olasd accepted this revision.Jun 19 2019, 2:15 PM

This revision is now accepted and ready to land.Jun 19 2019, 2:15 PM

Improve a bit JournalClient's docstring

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/172/ for more details.

Harbormaster completed remote builds in B6337: Diff 5356.Jun 19 2019, 2:35 PM

Closed by commit rDJNL253d150b5d0b: client: allow to specify any cfg parameter to KafkaConsumer() (authored by douardda). · Explain WhyJun 19 2019, 2:40 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Path

Size

swh/

journal/

cli.py

33 lines

client.py

6 lines

Diff 5359

View Options

swh/journal/cli.py