Page MenuHomeSoftware Heritage

journal client: add a test for real-world message processing
AcceptedPublic

Authored by olasd on Feb 1 2021, 2:11 PM.

Details

Summary

This adds a dump of the messages related to linux.git, and checks that
permutations of these messages yield sensible data in the origin visit
cache.

This explicitly resets the test fixture before every test to work around the
hypothesis limitation wrt function-scoped pytest fixtures.

I'm not sure we really want to land this.

Related to T3000

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18940
Build 29351: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 29350: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D4981 (id=17772)

Rebasing onto cf0583b079...

Current branch diff-target is up to date.
Changes applied before test
commit b7ec5ec0096d813265150f8b2195953ac1a55130
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Feb 1 12:37:23 2021 +0100

    journal client: add a test for real-world message processing
    
    This adds a dump of the messages related to linux.git, and checks that
    permutations of these messages yield sensible data in the origin visit
    cache.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/292/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/292/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 1 2021, 2:13 PM
Harbormaster failed remote builds in B18913: Diff 17772!

Properly override hypothesis health checks

Build is green

Patch application report for D4981 (id=17799)

Rebasing onto aaffff2631...

First, rewinding head to replay your work on top of it...
Applying: journal client: add a test for real-world message processing
Changes applied before test
commit 2d760f3404377e7070aad322f26e0faea4a6179d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Feb 1 12:37:23 2021 +0100

    journal client: add a test for real-world message processing
    
    This adds a dump of the messages related to linux.git, and checks that
    permutations of these messages yield sensible data in the origin visit
    cache.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/295/ for more details.

olasd requested review of this revision.Feb 2 2021, 11:10 AM

Cross-checking out of the db, and agreed on the test's output expectation.

I don't know if we want that test to be included as well.

Also, I'm surprised at the overall tests speed (around 65s only with that test [1]).
Last time we added a test with 720 permutations, it took quite a while to run
(but it was not with hypothesis, plain pytest.mark.parametrize and that was too
slow so we decreased the sample a lot).

Curious me, what does the strategies.permutations do, take a subset out of the
permutations and run the test on it?

[1] 259 passed, 1 xfailed in 66.15s (0:01:06) with that test that ran :)

swh/scheduler/tests/test_journal_client.py
1357

What does batches of 1 element give?

I'd expect it would reflect more what could have happened during the initial run... maybe?

Also, I'm surprised at the overall tests speed (around 65s only with that test [1]).
Last time we added a test with 720 permutations, it took quite a while to run
(but it was not with hypothesis, plain pytest.mark.parametrize and that was too
slow so we decreased the sample a lot).

Curious me, what does the strategies.permutations do, take a subset out of the
permutations and run the test on it?

Yes, it does the test on a limited random sample of permutations (like all random hypothesis tests, the sample is controlled by the hypothesis seed). If a test fails, hypothesis records the seed and the permutation that failed, and tries to reduce the sample to the smallest number of actual 1-1 permutations of lines.

swh/scheduler/tests/test_journal_client.py
1357

Exactly the same result, afaict.

Accepting it as i don't see any real blocker [1] on this being landed.

[1] aside maybe the test data being a bit hard to read through

This revision is now accepted and ready to land.Feb 4 2021, 10:40 AM

That probably can be closed since the journal client evolved since the time of this diff.