Remove the use of pytest-kafka
ClosedPublic
Actions

Authored by olasd on Apr 20 2020, 11:38 AM.

Details

Reviewers

douardda
ardumont

Group Reviewers

Reviewers

Commits

rDJNL9afb659fbc57: Replace pytest-kafka with the built-in confluent-kafka-python mock broker
rDJNL18518795e535: Replace expensive "Large directory" tests with a mock jogging the same codepath

Summary

Replace tests using the kafka configuration overrides (with tests using mocks instead), then drop pytest-kafka completely, in favor of the confluent-kafka-python mock broker.

Test Plan

tox -r -e py37

Diff Detail

Repository

rDJNL Journal infrastructure

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Apr 20 2020, 11:38 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 20 2020, 11:38 AM

Build has FAILED

Patch application report for D3030 (id=10762)

Rebasing onto b53d7d7c37...

Current branch diff-target is up to date.

Changes applied before test

commit 35ac146d63b4b54cb3242d9205ee6353e3bdf64d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Apr 14 12:08:11 2020 +0200

    Remove the use of pytest-kafka

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/48/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/48/console

Harbormaster failed remote builds in B11891: Diff 10762!Apr 20 2020, 11:43 AM

Replace expensive "Large directory" tests with a mock jogging the same codepath
improve commit messages

olasd requested review of this revision.Apr 22 2020, 12:57 PM

olasd edited the summary of this revision. (Show Details)

olasd edited the test plan for this revision. (Show Details)

Build is green

Patch application report for D3030 (id=10816)

Rebasing onto 913049fb0f...

Current branch diff-target is up to date.

Changes applied before test

commit 9afb659fbc57d7603c14c98190c7b7ef1a0704b6
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 22 12:54:41 2020 +0200

    Replace pytest-kafka with the built-in confluent-kafka-python mock broker
    
    This removes the dependency on java and a full kafka install for the tests of
    swh.journal, which should help keep integration tests in other split off
    components. They also feel a little bit faster (1m25s instead of 2m40s on this
    machine).

commit 18518795e535d9534fa379834dbc9d0ca0acf1e0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 22 12:32:55 2020 +0200

    Replace expensive "Large directory" tests with a mock jogging the same codepath
    
    The mocked kafka broker from librdkafka doesn't seem to honor message.max.bytes.
    We replace the tests depending on that feature with a mock which pretends that
    the delivery of messages always fails.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/49/ for more details.

Harbormaster completed remote builds in B11970: Diff 10816.Apr 22 2020, 12:59 PM