Page MenuHomeSoftware Heritage
Differential D2185

Consume messages from Kafka in batches of 20.
ClosedPublic
Actions

Authored by vlorentz on Oct 28 2019, 2:19 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDJNLc149860fdff4: Consume messages from Kafka in batches of 20.
Summary

Instead of one-by-one.
It's probably more efficient, and seems to partially mitigates T2034.

Depends on D2203

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Oct 28 2019, 2:19 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 28 2019, 2:19 PM
swh-public-ci added a comment.Oct 28 2019, 2:23 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/282/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/282/console

Harbormaster failed remote builds in B8648: Diff 7443!Oct 28 2019, 2:23 PM
vlorentz updated this revision to Diff 7445.Oct 28 2019, 3:16 PM

implement consume() in mock consumer.

swh-public-ci added a comment.Oct 28 2019, 3:24 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/284/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/284/console

Harbormaster failed remote builds in B8650: Diff 7445!Oct 28 2019, 3:24 PM
swh-public-ci added a comment.Oct 28 2019, 3:30 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/285/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/285/console

vlorentz updated this revision to Diff 7458.Oct 29 2019, 12:20 PM

rebase

vlorentz added a child revision: D2186: Close Consumer connection on replayer shut down..Oct 29 2019, 12:22 PM
swh-public-ci added a comment.Oct 29 2019, 12:38 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/290/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/290/console

Harbormaster failed remote builds in B8663: Diff 7458!Oct 29 2019, 12:38 PM
olasd accepted this revision.Oct 29 2019, 1:33 PM
This revision is now accepted and ready to land.Oct 29 2019, 1:33 PM
vlorentz updated this revision to Diff 7513.Oct 30 2019, 3:36 PM

Fix non-deterministic test failures + make them faster.

vlorentz requested review of this revision.Oct 30 2019, 3:37 PM
vlorentz edited the summary of this revision. (Show Details)
vlorentz added a parent revision: D2203: Add support for origin_visit['origin'] with type 'str'..

@olasd I made some major changes, please review again

vlorentz updated this revision to Diff 7514.Oct 30 2019, 3:37 PM

fix base commit

swh-public-ci added a comment.Oct 30 2019, 3:42 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/296/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/296/console

Harbormaster failed remote builds in B8718: Diff 7513!Oct 30 2019, 3:42 PM
vlorentz updated this revision to Diff 7516.Oct 30 2019, 3:43 PM

rebase

swh-public-ci added a comment.Oct 30 2019, 3:44 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/297/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/297/console

Harbormaster failed remote builds in B8719: Diff 7514!Oct 30 2019, 3:44 PM
swh-public-ci added a comment.Oct 30 2019, 3:49 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/299/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/299/console

Harbormaster failed remote builds in B8721: Diff 7516!Oct 30 2019, 3:49 PM
vlorentz updated this revision to Diff 7518.Oct 30 2019, 3:51 PM

rebase

swh-public-ci added a comment.Oct 30 2019, 3:55 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/301/ for more details.

Harbormaster completed remote builds in B8723: Diff 7518.Oct 30 2019, 3:55 PM
olasd accepted this revision.Oct 30 2019, 3:59 PM
This revision is now accepted and ready to land.Oct 30 2019, 3:59 PM
Closed by commit rDJNLc149860fdff4: Consume messages from Kafka in batches of 20. (authored by vlorentz). · Explain WhyOct 30 2019, 4:04 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDJNLc149860fdff4: Consume messages from Kafka in batches of 20..
vlorentz mentioned this in T2034: Unbreak journal clients.Nov 4 2019, 12:22 PM

Revision Contents
Changeset List

PathSize
swh/
journal/
5 lines
45 lines
tests/
2 lines
47 lines
6 lines

Diff 7523

View Options

swh/journal/cli.py

Loading...
View Options

swh/journal/client.py

Loading...
View Options

swh/journal/tests/test_replay.py

Loading...
View Options

swh/journal/tests/test_write_replay.py

Loading...
View Options

swh/journal/tests/utils.py

Loading...
About · Developer information · Legal