Page MenuHomeSoftware Heritage

hypothesis: Run with more examples by default
ClosedPublic

Authored by vlorentz on Aug 12 2021, 5:12 PM.

Details

Summary

Using a single example skips a lot of useful tests...
(eg. see D6081)

Diff Detail

Repository
rDWAPPS Web applications
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 23106
Build 36037: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 36036: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D6083 (id=22027)

Could not rebase; Attempt merge onto 70fb9a97c5...

Updating 70fb9a97..76ae8d0e
Fast-forward
 swh/web/api/views/metadata.py            | 16 ++----
 swh/web/tests/api/views/test_metadata.py | 97 ++++++++++++++++++++++----------
 swh/web/tests/api/views/utils.py         |  2 +
 swh/web/tests/conftest.py                |  2 +-
 4 files changed, 76 insertions(+), 41 deletions(-)
Changes applied before test
commit 76ae8d0ee6a27ba93f6a97fb9d50bd2e3e6ec802
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit a6eac2220a73f4a6e676218f80a64788dc6d7f95
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 15:38:05 2021 +0200

    Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile
    
    Hypothesis only runs the pagination tests with a single object when
    using the default profile...
    
    A future commit will change the default profile.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/994/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/994/console

Build has FAILED

Patch application report for D6083 (id=22050)

Could not rebase; Attempt merge onto c5259fce4f...

Updating c5259fce..823bddbd
Fast-forward
 swh/web/api/views/metadata.py            | 16 ++----
 swh/web/tests/api/views/test_metadata.py | 96 ++++++++++++++++++++++----------
 swh/web/tests/api/views/utils.py         |  2 +
 swh/web/tests/conftest.py                |  2 +-
 4 files changed, 75 insertions(+), 41 deletions(-)
Changes applied before test
commit 823bddbd2062596a43f6a06523a579ca10713a0e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit 48e9258f8d10b2983b5ff6d24900bd04c80e02b4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 15:38:05 2021 +0200

    Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile
    
    Hypothesis only runs the pagination tests with a single object when
    using the default profile...
    
    A future commit will change the default profile.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/998/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/998/console

I need help to fix these failures...

I started investigating the numerous failures when running the tests with multiple examples (make test-full run tests with default hypothesis number of examples).

I managed to fix a couple of them by ensuring strategies do not depend on external states but the main issue is the archive_data pytest fixture not reseted
between the test runs for the different hypothesis examples. One possible solution is to use pytest-subtesthack to ensure fixtures are reseted between examples.
Based on my first tests, it seems to work as expected. Tests that modify the test archive content (creating origin visits for instance) should be adapted to ensure
they are stateless.

Sounds good, thanks! Should I close this diff?

Sounds good, thanks! Should I close this diff?

Let's keep it open until I submit my diff on the subject.

@vlorentz, I have submitted D6115 and D6116 to ensure swh-web tests can be safely executed with multiple hypothesis examples.

You can rebase that diff once it is landed to bump number of examples in fast hypothesis profile.

Build is green

Patch application report for D6083 (id=22149)

Rebasing onto c7548f93a1...

Current branch diff-target is up to date.
Changes applied before test
commit b94e7f7638a15f56db04a8e5a0db9df8463ef673
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1016/ for more details.

This revision is now accepted and ready to land.Aug 23 2021, 5:17 PM

Build is green

Patch application report for D6083 (id=22231)

Could not rebase; Attempt merge onto a45155af91...

Updating a45155af..dc103921
Fast-forward
 requirements-swh.txt      | 2 +-
 swh/web/tests/conftest.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Changes applied before test
commit dc10392144a038a611cf409fd1def1c30a19f2a4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit 66865c37ce33ae6ac33ca11737ef0140c735a7a0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 26 12:00:13 2021 +0200

    Bump minimum swh-vault version
    
    v1.0.0 changed the interface and we need the new one.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1035/ for more details.

This revision was landed with ongoing or failed builds.Aug 27 2021, 2:30 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D6083 (id=22274)

Rebasing onto 3a448d88c8...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-1039-D6083.
Changes applied before test

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1039/ for more details.