Paths

Table of Contentst

Differential D6115

tests: Ensure they all can be run with multiple hypothesis examples
ClosedPublic
Actions

Authored by anlambert on Aug 19 2021, 12:12 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T1695: Make hypothesis strategies for swh-web stateless

Commits

rDWAPPS87cc9e042dc2: tests: Ensure they all can be run with multiple hypothesis examples

Summary

When running swh-web tests using make test-full, multiple hypothesis examples
are provided as test inputs instead of a single one when running make test.
In that case some tests were failing mostly due to the fact they were not stateless
between test runs.

That commit fixes those tests execution and ensures stateless test runs by:

turning some hypothesis strategies into stateless ones
turning the archive_data fixture into a function scope one
using subtest fixture from pytest-subtesthack when it is required to reset the archive_data fixture for a test between hypothesis example runs

As a consequence, tests will be longer to execute as global state will be reseted
between each test.

Nevertheless, metadata related tests are still failing when running with multiple
hypothesis examples, fix will be handled in next commit.

Related to T1695

Diff Detail

Repository

rDWAPPS Web applications

Branch

hypothesis-multi-examples-fix

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 23079
Build 35990: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 35989: arc lint + arc unit

Event Timeline

anlambert created this revision.Aug 19 2021, 12:12 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 19 2021, 12:12 PM

anlambert added a child revision: D6116: api/metadata: Fix issues detected with hypothesis.Aug 19 2021, 12:12 PM

anlambert mentioned this in D6083: hypothesis: Run with more examples by default.Aug 19 2021, 12:15 PM

Build is green

Patch application report for D6115 (id=22120)

Rebasing onto 885be5dde5...

Current branch diff-target is up to date.

Changes applied before test

commit fc086320d520c5f6ecc4411ac5fd4b2b71bcbdad
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Aug 19 12:00:40 2021 +0200

    tests: Ensure they all can be run with multiple hypothesis examples
    
    When running swh-web tests using 'make test-full', multiple hypothesis examples
    are provided as test inputs instead of a single one when running 'make test'.
    In that case some tests were failing mostly due to the fact they were not stateless
    between test runs.
    
    That commit fixes those tests execution and ensures stateless test runs by:
    
      - turning some hypothesis strategies into stateless ones
    
      - turning the archive_data fixture into a function scope one
    
      - using subtest fixture from pytest-subtesthack when it is required to reset
        the archive_data fixture for a test between hypothesis example runs
    
    As a consequence, tests will be longer to execute as global state will be reseted
    between each test.
    
    Nevertheless, metadata related tests are still failing when running with multiple
    hypothesis examples, fix will be handled in next commit.
    
    Related to T1695

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1007/ for more details.

Harbormaster completed remote builds in B23077: Diff 22120.Aug 19 2021, 12:27 PM

anlambert requested review of this revision.Aug 19 2021, 12:27 PM

anlambert added inline comments.Aug 19 2021, 12:51 PM

swh/web/tests/browse/views/test_origin.py
646	lookup_snapshot_sizes is called once and result is fetched from cache next times so that assertion do not pass with multiple hypothesis examples
699	same here

anlambert mentioned this in D6081: Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile.Aug 19 2021, 1:01 PM

Fix a comment

Build is green

Patch application report for D6115 (id=22122)

Rebasing onto 885be5dde5...

Current branch diff-target is up to date.

Changes applied before test

commit 87cc9e042dc2e3cc79c63c25b0bd5f04484693b4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Aug 19 12:00:40 2021 +0200

    tests: Ensure they all can be run with multiple hypothesis examples
    
    When running swh-web tests using 'make test-full', multiple hypothesis examples
    are provided as test inputs instead of a single one when running 'make test'.
    In that case some tests were failing mostly due to the fact they were not stateless
    between test runs.
    
    That commit fixes those tests execution and ensures stateless test runs by:
    
      - turning some hypothesis strategies into stateless ones
    
      - turning the archive_data fixture into a function scope one
    
      - using subtest fixture from pytest-subtesthack when it is required to reset
        the archive_data fixture for a test between hypothesis example runs
    
    As a consequence, tests will be longer to execute as global state will be reseted
    between each test.
    
    Nevertheless, metadata related tests are still failing when running with multiple
    hypothesis examples, fix will be handled in next commit.
    
    Related to T1695

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1009/ for more details.

Harbormaster completed remote builds in B23079: Diff 22122.Aug 19 2021, 1:19 PM

Does it mean we need to do def test_inner on *every* test that uses hypothesis?

swh/web/tests/conftest.py
422–424	unused?

I really can't find a better way to do this :(

Although we should probably replace hypothesis with pytest.mark.parametrize at some point, it seems to be less trouble.

This revision is now accepted and ready to land.Aug 19 2021, 1:53 PM

In D6115#157987, @vlorentz wrote:

Does it mean we need to do def test_inner on *every* test that uses hypothesis?

Fortunately not, almost all tests can be safely executed with multiple hypothesis examples.
There was only a couple that required that type of hack on the pytest fixtures.

Although we should probably replace hypothesis with pytest.mark.parametrize at some point, it seems to be less trouble.

Hypothesis slow downs tests and they are hard to debug sometimes but it still help uncovering bugs so not sure about dropping its use.

swh/web/tests/conftest.py
422–424	I just copied pasted the code from pytest-subtesthack, looks like that path is not taken in our current tests but it must be there for a reason.