Page MenuHomeSoftware Heritage
Differential D6083

hypothesis: Run with more examples by default
ClosedPublic
Actions

Authored by vlorentz on Aug 12 2021, 5:12 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDWAPPS3a448d88c80f: hypothesis: Run with more examples by default
Summary

Using a single example skips a lot of useful tests...
(eg. see D6081)

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Aug 12 2021, 5:12 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 12 2021, 5:12 PM
vlorentz published this revision for review.Aug 12 2021, 5:12 PM
vlorentz mentioned this in D6081: Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile.
vlorentz added a parent revision: D6081: Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile.

I need help to fix these failures...

vlorentz added a reviewer: anlambert.Aug 12 2021, 5:13 PM
swh-public-ci added a comment.Aug 12 2021, 5:39 PM

Build has FAILED

Patch application report for D6083 (id=22027)

Could not rebase; Attempt merge onto 70fb9a97c5...

Updating 70fb9a97..76ae8d0e
Fast-forward
 swh/web/api/views/metadata.py            | 16 ++----
 swh/web/tests/api/views/test_metadata.py | 97 ++++++++++++++++++++++----------
 swh/web/tests/api/views/utils.py         |  2 +
 swh/web/tests/conftest.py                |  2 +-
 4 files changed, 76 insertions(+), 41 deletions(-)
Changes applied before test
commit 76ae8d0ee6a27ba93f6a97fb9d50bd2e3e6ec802
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit a6eac2220a73f4a6e676218f80a64788dc6d7f95
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 15:38:05 2021 +0200

    Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile
    
    Hypothesis only runs the pagination tests with a single object when
    using the default profile...
    
    A future commit will change the default profile.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/994/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/994/console

Harbormaster failed remote builds in B22984: Diff 22027!Aug 12 2021, 5:39 PM
vlorentz updated this revision to Diff 22050.Aug 16 2021, 9:49 AM

rebase

swh-public-ci added a comment.Aug 16 2021, 10:18 AM

Build has FAILED

Patch application report for D6083 (id=22050)

Could not rebase; Attempt merge onto c5259fce4f...

Updating c5259fce..823bddbd
Fast-forward
 swh/web/api/views/metadata.py            | 16 ++----
 swh/web/tests/api/views/test_metadata.py | 96 ++++++++++++++++++++++----------
 swh/web/tests/api/views/utils.py         |  2 +
 swh/web/tests/conftest.py                |  2 +-
 4 files changed, 75 insertions(+), 41 deletions(-)
Changes applied before test
commit 823bddbd2062596a43f6a06523a579ca10713a0e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit 48e9258f8d10b2983b5ff6d24900bd04c80e02b4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 15:38:05 2021 +0200

    Fix api_raw_extrinsic_metadata_swhid-related bugs found when using the 'slow' hypothesis profile
    
    Hypothesis only runs the pagination tests with a single object when
    using the default profile...
    
    A future commit will change the default profile.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/998/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/998/console

Harbormaster failed remote builds in B23006: Diff 22050!Aug 16 2021, 10:18 AM
anlambert added a comment.Aug 18 2021, 3:35 PM

I need help to fix these failures...

I started investigating the numerous failures when running the tests with multiple examples (make test-full run tests with default hypothesis number of examples).

I managed to fix a couple of them by ensuring strategies do not depend on external states but the main issue is the archive_data pytest fixture not reseted
between the test runs for the different hypothesis examples. One possible solution is to use pytest-subtesthack to ensure fixtures are reseted between examples.
Based on my first tests, it seems to work as expected. Tests that modify the test archive content (creating origin visits for instance) should be adapted to ensure
they are stateless.

vlorentz added a comment.Aug 18 2021, 3:37 PM

Sounds good, thanks! Should I close this diff?

anlambert added a comment.Aug 18 2021, 3:40 PM
In D6083#157897, @vlorentz wrote:

Sounds good, thanks! Should I close this diff?

Let's keep it open until I submit my diff on the subject.

anlambert added a comment.Aug 19 2021, 12:15 PM

@vlorentz, I have submitted D6115 and D6116 to ensure swh-web tests can be safely executed with multiple hypothesis examples.

You can rebase that diff once it is landed to bump number of examples in fast hypothesis profile.

vlorentz updated this revision to Diff 22149.Aug 23 2021, 5:03 PM

rebase

swh-public-ci added a comment.Aug 23 2021, 5:17 PM

Build is green

Patch application report for D6083 (id=22149)

Rebasing onto c7548f93a1...

Current branch diff-target is up to date.
Changes applied before test
commit b94e7f7638a15f56db04a8e5a0db9df8463ef673
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1016/ for more details.

Harbormaster completed remote builds in B23106: Diff 22149.Aug 23 2021, 5:17 PM
anlambert accepted this revision.Aug 23 2021, 5:17 PM
This revision is now accepted and ready to land.Aug 23 2021, 5:17 PM
vlorentz updated this revision to Diff 22231.Aug 26 2021, 12:14 PM

rebase

swh-public-ci added a comment.Aug 26 2021, 12:38 PM

Build is green

Patch application report for D6083 (id=22231)

Could not rebase; Attempt merge onto a45155af91...

Updating a45155af..dc103921
Fast-forward
 requirements-swh.txt      | 2 +-
 swh/web/tests/conftest.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
Changes applied before test
commit dc10392144a038a611cf409fd1def1c30a19f2a4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 12 17:12:06 2021 +0200

    hypothesis: Run with more examples by default
    
    Using a single example skips a lot of useful tests...

commit 66865c37ce33ae6ac33ca11737ef0140c735a7a0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 26 12:00:13 2021 +0200

    Bump minimum swh-vault version
    
    v1.0.0 changed the interface and we need the new one.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1035/ for more details.

Harbormaster completed remote builds in B23189: Diff 22231.Aug 26 2021, 12:39 PM
vlorentz updated this revision to Diff 22274.Aug 27 2021, 2:30 PM

rebase

This revision was landed with ongoing or failed builds.Aug 27 2021, 2:30 PM
Closed by commit rDWAPPS3a448d88c80f: hypothesis: Run with more examples by default (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDWAPPS3a448d88c80f: hypothesis: Run with more examples by default.
swh-public-ci added a comment.Aug 27 2021, 2:44 PM

Build is green

Patch application report for D6083 (id=22274)

Rebasing onto 3a448d88c8...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-1039-D6083.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1039/ for more details.
Harbormaster completed remote builds in B23231: Diff 22274.Aug 27 2021, 2:44 PM
Revision Contents
Changeset List
PathSize
swh/
web/
tests/
2 lines
Diff 22275
View Options
swh/web/tests/conftest.py
Loading...
About · Developer information · Legal