Page MenuHomeSoftware Heritage
Differential D2979

Allow overriding broker settings in the kafka server fixture
ClosedPublic
Actions

Authored by olasd on Apr 8 2020, 1:36 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDJNLbf7908bcaa95: Allow overriding broker settings in the kafka server fixture
Summary

To test the behavior of swh.journal under different settings of the kafka
broker, split the kafka_server fixture into:

  • a persistent session_kafka_server fixture, spawning a single broker for the whole test session
  • a function-scoped kafka_server fixture, which uses the kafka admin client to override broker settings if needed, for the current test (and restores the original setting values afterwards)

The broker settings can be overridden for a given function using the parametrize
decorator to change the return value of the kafka_server_config_overrides
fixture.

Test Plan

tox is still happy

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd created this revision.Apr 8 2020, 1:36 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 8 2020, 1:36 PM
swh-public-ci added a comment.Apr 8 2020, 1:57 PM

Build was aborted

Patch application report for D2979 (id=10588)

Rebasing onto 17d64ff61b...

Current branch diff-target is up to date.
Changes applied before test
commit ba13209e069f57d41b21f97e5c2c8d308127f631
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 13:15:45 2020 +0200

    Allow overriding broker settings in the kafka server fixture
    
    To test the behavior of swh.journal under different settings of the kafka
    broker, split the kafka_server fixture into:
    
     - a persistent session_kafka_server fixture, spawning a single broker for the
     whole test session
    
     - a function-scoped kafka_server fixture, which uses the kafka admin client to
     override broker settings if needed, for the current test (and restores the
     original setting values afterwards)
    
    The broker settings can be overridden for a given function using the parametrize
    decorator to change the return value of the kafka_server_config_overrides
    fixture.

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/17/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/17/console

Harbormaster failed remote builds in B11675: Diff 10588!Apr 8 2020, 1:57 PM
ardumont added a subscriber: ardumont.Apr 8 2020, 2:43 PM

"tox is still happy" but timeouts in the jenkins build ;)

swh/journal/tests/conftest.py
282

Shouldn't we log something here?

At least to ease introspection in build failures if something goes completely out of control at some point?

(same goes for the other try except below)

ardumont accepted this revision.Apr 8 2020, 2:44 PM

Jokes/Question aside, looks good.

This revision is now accepted and ready to land.Apr 8 2020, 2:44 PM
swh-public-ci added a comment.Apr 8 2020, 2:45 PM

Build is green

Patch application report for D2979 (id=10588)

Rebasing onto a68e800622...

First, rewinding head to replay your work on top of it...
Applying: Allow overriding broker settings in the kafka server fixture
Changes applied before test
commit e0a214d284fdc6684b0961079c28b5e06abc1d0a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 13:15:45 2020 +0200

    Allow overriding broker settings in the kafka server fixture
    
    To test the behavior of swh.journal under different settings of the kafka
    broker, split the kafka_server fixture into:
    
     - a persistent session_kafka_server fixture, spawning a single broker for the
     whole test session
    
     - a function-scoped kafka_server fixture, which uses the kafka admin client to
     override broker settings if needed, for the current test (and restores the
     original setting values afterwards)
    
    The broker settings can be overridden for a given function using the parametrize
    decorator to change the return value of the kafka_server_config_overrides
    fixture.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/19/ for more details.

Harbormaster completed remote builds in B11675: Diff 10588.Apr 8 2020, 2:45 PM
olasd marked an inline comment as done.Apr 8 2020, 2:50 PM

Well, the "swh.journal tests timeout randomly" bug is still here, I guess.

swh/journal/tests/conftest.py
282

I think you misread the construct: the exception gets re-raised. (I'm just making the "processing" of the concurrent.future explicit here).

olasd updated this revision to Diff 10591.Apr 8 2020, 2:51 PM
olasd marked an inline comment as done.

Rebase.

swh-public-ci added a comment.Apr 8 2020, 2:56 PM

Build is green

Patch application report for D2979 (id=10591)

Rebasing onto a68e800622...

Current branch diff-target is up to date.
Changes applied before test
commit bf7908bcaa95cd121b9896d109240a6fb63fe0cf
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Apr 8 13:15:45 2020 +0200

    Allow overriding broker settings in the kafka server fixture
    
    To test the behavior of swh.journal under different settings of the kafka
    broker, split the kafka_server fixture into:
    
     - a persistent session_kafka_server fixture, spawning a single broker for the
     whole test session
    
     - a function-scoped kafka_server fixture, which uses the kafka admin client to
     override broker settings if needed, for the current test (and restores the
     original setting values afterwards)
    
    The broker settings can be overridden for a given function using the parametrize
    decorator to change the return value of the kafka_server_config_overrides
    fixture.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/20/ for more details.

Harbormaster completed remote builds in B11678: Diff 10591.Apr 8 2020, 2:56 PM
ardumont added inline comments.Apr 8 2020, 2:57 PM
swh/journal/tests/conftest.py
282

right.

Closed by commit rDJNLbf7908bcaa95: Allow overriding broker settings in the kafka server fixture (authored by olasd). · Explain WhyApr 8 2020, 2:57 PM
This revision was automatically updated to reflect the committed changes.
olasd added a commit: rDJNLbf7908bcaa95: Allow overriding broker settings in the kafka server fixture.

Revision Contents
Changeset List

PathSize
swh/
journal/
tests/
88 lines

Diff 10592

View Options

swh/journal/tests/conftest.py

Loading...
About · Developer information · Legal