HomeSoftware Heritage

tests: Ensure failing test with random fixture data can be reproduced

Description

tests: Ensure failing test with random fixture data can be reproduced

Some tests input data returned by swh-web fixtures are sampled from
filtered objects stored in a static test archive.

In order to reproduce any failing test with the same fixture data:

  • add a new autouse function scope fixture setting the random seed to current time before each test execution and putting that seed in cache
  • implement custom pytest reporting which adds a new section detailing the random seeds used for each failing test and the pytests commands to reproduce the failures
  • add a new pytest option --swh-web-random-seed enabling to explicitly specify the random seed that will be used to run the tests

Details

Provenance
anlambertAuthored on Oct 14 2021, 3:33 PM
anlambertPushed on Oct 15 2021, 4:40 PM
Differential Revision
D6476: tests: Ensure failing test with random fixture data can be reproduced
Parents
rDWAPPSefbf0cdc2c4d: tests/conftest: Put filtered input data in cache
Branches
Unknown
Tags
Unknown
Build Status
Buildable 24473
Build 38185: test-and-buildJenkins console · Jenkins