Page MenuHomeSoftware Heritage

tests: Ensure failing test with random fixture data can be reproduced
ClosedPublic

Authored by anlambert on Oct 14 2021, 3:46 PM.

Details

Summary

Some tests input data returned by swh-web fixtures are sampled from
filtered objects stored in a static test archive.

In order to reproduce any failing test with the same fixture data:

  • add a new autouse function scope fixture setting the random seed to current time before each test execution and putting that seed in cache
  • implement custom pytest reporting which adds a new section detailing the random seeds used for each failing test and the pytests commands to reproduce the failures
  • add a new pytest option --swh-web-random-seed enabling to explicitly specify the random seed that will be used to run the tests

Depends on D6471

After hacking on a test to make it fail, this is the pytest output we get:

15:39 $ pytest -sv swh/web/tests/api/views/test_directory.py::test_api_directory
================================================================================================================================== test session starts ==================================================================================================================================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/anlambert/.virtualenvs/swh/bin/python
cachedir: .pytest_cache
django: settings: swh.web.settings.tests (from ini)
hypothesis profile 'swh-web-fast' -> deadline=None, max_examples=5, database=DirectoryBasedExampleDatabase('/home/anlambert/swh/swh-environment/swh-web/.hypothesis/examples'), suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much, HealthCheck.function_scoped_fixture]
rootdir: /home/anlambert/swh/swh-environment/swh-web, configfile: pytest.ini
plugins: django-4.4.0, postgresql-3.1.1, hypothesis-6.23.2, mock-3.6.1, requests-mock-1.9.3, xdist-2.4.0, redis-2.1.1, dash-2.0.0, forked-1.3.0, asyncio-0.15.1, django-test-migrations-1.1.0, swh.core-0.15.0, swh.journal-0.8.1.dev1+g95d945e
collected 1 item                                                                                                                                                                                                                                                                        

swh/web/tests/api/views/test_directory.py::test_api_directory FAILED

======================================================================================================================================= FAILURES ========================================================================================================================================
__________________________________________________________________________________________________________________________________ test_api_directory ___________________________________________________________________________________________________________________________________

api_client = <rest_framework.test.APIClient object at 0x7ff03702a910>, archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7ff03702a7f0>, directory = 'bf0bc5553f663cf2d58c6b86a0bf409e23eb60ee'

    def test_api_directory(api_client, archive_data, directory):
    
        url = reverse("api-1-directory", url_args={"sha1_git": directory})
        rv = check_api_get_responses(api_client, url, status_code=200)
    
        dir_content = list(archive_data.directory_ls(directory))
        expected_data = list(
            map(enrich_directory_entry, dir_content, [rv.wsgi_request] * len(dir_content))
        )
    
>       assert rv.data != expected_data
E       assert [{'checksums': {'sha1': 'a64431fdd98a365de8804650801fcf09e6e1fa25', 'sha1_git': '7895cfc379e8ac385d9e8690834d0e0396d88...8'}, 'dir_id': 'bf0bc5553f663cf2d58c6b86a0bf409e23eb60ee', 'length': 5353, 'name': 'highlightjs-line-numbers.js', ...}] != [{'checksums': {'sha1': 'a64431fdd98a365de8804650801fcf09e6e1fa25', 'sha1_git': '7895cfc379e8ac385d9e8690834d0e0396d88...8'}, 'dir_id': 'bf0bc5553f663cf2d58c6b86a0bf409e23eb60ee', 'length': 5353, 'name': 'highlightjs-line-numbers.js', ...}]
E        +  where [{'checksums': {'sha1': 'a64431fdd98a365de8804650801fcf09e6e1fa25', 'sha1_git': '7895cfc379e8ac385d9e8690834d0e0396d88...8'}, 'dir_id': 'bf0bc5553f663cf2d58c6b86a0bf409e23eb60ee', 'length': 5353, 'name': 'highlightjs-line-numbers.js', ...}] = <Response status_code=200, "application/json">.data

swh/web/tests/api/views/test_directory.py:24: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------------------------------------------------------
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/memononen/libtess2' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/memononen/libtess2 - Uncompressing archive libtess2.zip at /tmp/tmp.4lxzsfz5.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/memononen/libtess2 - Done injecting /tmp/tmp.4lxzsfz5.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js.zip at /tmp/tmp.xmkex_j3.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.xmkex_j3.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js_visit2.zip at /tmp/tmp.lpiu0lga.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.lpiu0lga.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'repo_with_submodules' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project repo_with_submodules - Uncompressing archive repo_with_submodules.tgz at /tmp/tmp.cj3vdywt.swh.loader.git/repo_with_submodules
INFO     swh.loader.git.from_disk:from_disk.py:448 Project repo_with_submodules - Done injecting /tmp/tmp.cj3vdywt.swh.loader.git/repo_with_submodules
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.mimetype.MimetypeIndexer object at 0x7ff032079eb0>: config={'indexer_storage': {'cls': 'memory', 'args': {}}, 'tools': {'name': 'file', 'version': '1:5.30-1+deb9u1', 'configuration': {'type': 'library', 'debian-package': 'python3-magic'}}, 'storage': {'cls': 'memory'}, 'objstorage': {'cls': 'memory', 'args': {}}}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.fossology_license.FossologyLicenseIndexer object at 0x7ff01aa435e0>: config={'tools': {'name': 'nomos', 'version': '3.1.0rc2-31-ga2cbb8c', 'configuration': {'command_line': 'nomossa <filepath>'}}, 'workdir': '/tmp/swh/indexer.fossology.license', 'storage': {'cls': 'memory'}, 'objstorage': {'cls': 'memory', 'args': {}}, 'indexer_storage': {'cls': 'memory', 'args': {}}}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.ctags.CtagsIndexer object at 0x7ff03b14a220>: config={'tools': {'name': 'universal-ctags', 'version': '~git7859817b', 'configuration': {'command_line': 'ctags --fields=+lnz --sort=no --links=no --output-format=json <filepath>'}}, 'languages': {'c': 'c'}, 'workdir': '/tmp/swh/indexer.ctags', 'storage': {'cls': 'memory'}, 'objstorage': {'cls': 'memory', 'args': {}}, 'indexer_storage': {'cls': 'memory', 'args': {}}}
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218842 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
=================================================================================================================================== warnings summary ====================================================================================================================================
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/factory.py:80: DeprecationWarning: Explicit "args" key is deprecated for objstorage initialization, use class arguments keys directly instead.
    warnings.warn(

swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-indexer/swh/indexer/storage/__init__.py:68: DeprecationWarning: Explicit "args" key is deprecated, use keys directly instead.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218842 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
================================================================================================================================ short test summary info ================================================================================================================================
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory - assert [{'checksums': {'sha1': 'a64431fdd98a365de8804650801fcf09e6e1fa25', 'sha1_git': '7895cfc379e8ac385d9e8690834d0e0396d88...8'}, 'dir_id': 'bf0bc5553f663cf2d58c6b86a0bf409e23eb60ee', 'length': 5353, 'nam...
============================================================================================================================= 1 failed, 6 warnings in 2.78s =============================================================================================================================

If we run the same test again, we can see that directory fixture value is not the same due to random sampling:

15:40 $ pytest -sv swh/web/tests/api/views/test_directory.py::test_api_directory
================================================================================================================================== test session starts ==================================================================================================================================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/anlambert/.virtualenvs/swh/bin/python
cachedir: .pytest_cache
django: settings: swh.web.settings.tests (from ini)
hypothesis profile 'swh-web-fast' -> deadline=None, max_examples=5, database=DirectoryBasedExampleDatabase('/home/anlambert/swh/swh-environment/swh-web/.hypothesis/examples'), suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much, HealthCheck.function_scoped_fixture]
rootdir: /home/anlambert/swh/swh-environment/swh-web, configfile: pytest.ini
plugins: django-4.4.0, postgresql-3.1.1, hypothesis-6.23.2, mock-3.6.1, requests-mock-1.9.3, xdist-2.4.0, redis-2.1.1, dash-2.0.0, forked-1.3.0, asyncio-0.15.1, django-test-migrations-1.1.0, swh.core-0.15.0, swh.journal-0.8.1.dev1+g95d945e
collected 1 item                                                                                                                                                                                                                                                                        

swh/web/tests/api/views/test_directory.py::test_api_directory FAILED

======================================================================================================================================= FAILURES ========================================================================================================================================
__________________________________________________________________________________________________________________________________ test_api_directory ___________________________________________________________________________________________________________________________________

api_client = <rest_framework.test.APIClient object at 0x7fcf183a58e0>, archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7fcf183a57c0>, directory = 'cd19126d815470b28919d64b2a8e6a3e37f900dd'

    def test_api_directory(api_client, archive_data, directory):
    
        url = reverse("api-1-directory", url_args={"sha1_git": directory})
        rv = check_api_get_responses(api_client, url, status_code=200)
    
        dir_content = list(archive_data.directory_ls(directory))
        expected_data = list(
            map(enrich_directory_entry, dir_content, [rv.wsgi_request] * len(dir_content))
        )
    
>       assert rv.data != expected_data
E       assert [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...] != [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...]
E        +  where [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...] = <Response status_code=200, "application/json">.data

swh/web/tests/api/views/test_directory.py:24: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------------------------------------------------------
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/memononen/libtess2' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/memononen/libtess2 - Uncompressing archive libtess2.zip at /tmp/tmp.yexmaku6.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/memononen/libtess2 - Done injecting /tmp/tmp.yexmaku6.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js.zip at /tmp/tmp.1mnjhyyj.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.1mnjhyyj.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js_visit2.zip at /tmp/tmp.vqfk0ph8.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.vqfk0ph8.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'repo_with_submodules' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project repo_with_submodules - Uncompressing archive repo_with_submodules.tgz at /tmp/tmp.y480bwjf.swh.loader.git/repo_with_submodules
INFO     swh.loader.git.from_disk:from_disk.py:448 Project repo_with_submodules - Done injecting /tmp/tmp.y480bwjf.swh.loader.git/repo_with_submodules
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.mimetype.MimetypeIndexer object at 0x7fcf133fa310>: config={'tools': {'name': 'file', 'configuration': {'type': 'library', 'debian-package': 'python3-magic'}, 'version': '1:5.30-1+deb9u1'}, 'indexer_storage': {'cls': 'memory', 'args': {}}, 'storage': {'cls': 'memory'}, 'objstorage': {'cls': 'memory', 'args': {}}}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.fossology_license.FossologyLicenseIndexer object at 0x7fcefbcf17f0>: config={'workdir': '/tmp/swh/indexer.fossology.license', 'storage': {'cls': 'memory'}, 'tools': {'name': 'nomos', 'configuration': {'command_line': 'nomossa <filepath>'}, 'version': '3.1.0rc2-31-ga2cbb8c'}, 'indexer_storage': {'cls': 'memory', 'args': {}}, 'objstorage': {'cls': 'memory', 'args': {}}}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.ctags.CtagsIndexer object at 0x7fcf1c4c4220>: config={'workdir': '/tmp/swh/indexer.ctags', 'storage': {'cls': 'memory'}, 'tools': {'name': 'universal-ctags', 'configuration': {'command_line': 'ctags --fields=+lnz --sort=no --links=no --output-format=json <filepath>'}, 'version': '~git7859817b'}, 'indexer_storage': {'cls': 'memory', 'args': {}}, 'languages': {'c': 'c'}, 'objstorage': {'cls': 'memory', 'args': {}}}
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218902 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
=================================================================================================================================== warnings summary ====================================================================================================================================
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/factory.py:80: DeprecationWarning: Explicit "args" key is deprecated for objstorage initialization, use class arguments keys directly instead.
    warnings.warn(

swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-indexer/swh/indexer/storage/__init__.py:68: DeprecationWarning: Explicit "args" key is deprecated, use keys directly instead.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218902 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
================================================================================================================================ short test summary info ================================================================================================================================
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory - assert [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'len...
============================================================================================================================= 1 failed, 6 warnings in 2.76s =============================================================================================================================

If we explicitely pass the random seed used in previous test run, we can see the directory fixture value is now the same:

15:41 $ pytest -sv --swh-web-random-seed=1634218902 swh/web/tests/api/views/test_directory.py::test_api_directory
================================================================================================================================== test session starts ==================================================================================================================================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/anlambert/.virtualenvs/swh/bin/python
cachedir: .pytest_cache
django: settings: swh.web.settings.tests (from ini)
hypothesis profile 'swh-web-fast' -> deadline=None, max_examples=5, database=DirectoryBasedExampleDatabase('/home/anlambert/swh/swh-environment/swh-web/.hypothesis/examples'), suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much, HealthCheck.function_scoped_fixture]
rootdir: /home/anlambert/swh/swh-environment/swh-web, configfile: pytest.ini
plugins: django-4.4.0, postgresql-3.1.1, hypothesis-6.23.2, mock-3.6.1, requests-mock-1.9.3, xdist-2.4.0, redis-2.1.1, dash-2.0.0, forked-1.3.0, asyncio-0.15.1, django-test-migrations-1.1.0, swh.core-0.15.0, swh.journal-0.8.1.dev1+g95d945e
collected 1 item                                                                                                                                                                                                                                                                        

swh/web/tests/api/views/test_directory.py::test_api_directory FAILED

======================================================================================================================================= FAILURES ========================================================================================================================================
__________________________________________________________________________________________________________________________________ test_api_directory ___________________________________________________________________________________________________________________________________

api_client = <rest_framework.test.APIClient object at 0x7fd6bf2211f0>, archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7fd6bf23b850>, directory = 'cd19126d815470b28919d64b2a8e6a3e37f900dd'

    def test_api_directory(api_client, archive_data, directory):
    
        url = reverse("api-1-directory", url_args={"sha1_git": directory})
        rv = check_api_get_responses(api_client, url, status_code=200)
    
        dir_content = list(archive_data.directory_ls(directory))
        expected_data = list(
            map(enrich_directory_entry, dir_content, [rv.wsgi_request] * len(dir_content))
        )
    
>       assert rv.data != expected_data
E       assert [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...] != [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...]
E        +  where [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'length': 3479, 'name': 'geom.h', ...}, ...] = <Response status_code=200, "application/json">.data

swh/web/tests/api/views/test_directory.py:24: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured log setup -----------------------------------------------------------------------------------------------------------------------------------
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/memononen/libtess2' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/memononen/libtess2 - Uncompressing archive libtess2.zip at /tmp/tmp.y6zjknjo.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/memononen/libtess2 - Done injecting /tmp/tmp.y6zjknjo.swh.loader.git/libtess2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js.zip at /tmp/tmp.jcmzrsw7.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.jcmzrsw7.swh.loader.git/highlightjs-line-numbers.js
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'https://github.com/wcoder/highlightjs-line-numbers.js' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project https://github.com/wcoder/highlightjs-line-numbers.js - Uncompressing archive highlightjs-line-numbers.js_visit2.zip at /tmp/tmp.l6g4nvlt.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk:from_disk.py:448 Project https://github.com/wcoder/highlightjs-line-numbers.js - Done injecting /tmp/tmp.l6g4nvlt.swh.loader.git/highlightjs-line-numbers.js_visit2
INFO     swh.loader.git.from_disk.GitLoaderFromArchive:loader.py:330 Load origin 'repo_with_submodules' with type 'git'
INFO     swh.loader.git.from_disk:from_disk.py:433 Project repo_with_submodules - Uncompressing archive repo_with_submodules.tgz at /tmp/tmp.w7dwwecu.swh.loader.git/repo_with_submodules
INFO     swh.loader.git.from_disk:from_disk.py:448 Project repo_with_submodules - Done injecting /tmp/tmp.w7dwwecu.swh.loader.git/repo_with_submodules
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.mimetype.MimetypeIndexer object at 0x7fd6ba290eb0>: config={'tools': {'configuration': {'debian-package': 'python3-magic', 'type': 'library'}, 'name': 'file', 'version': '1:5.30-1+deb9u1'}, 'indexer_storage': {'args': {}, 'cls': 'memory'}, 'storage': {'cls': 'memory'}, 'objstorage': {'args': {}, 'cls': 'memory'}}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.fossology_license.FossologyLicenseIndexer object at 0x7fd6a2c559d0>: config={'tools': {'configuration': {'command_line': 'nomossa <filepath>'}, 'name': 'nomos', 'version': '3.1.0rc2-31-ga2cbb8c'}, 'storage': {'cls': 'memory'}, 'objstorage': {'args': {}, 'cls': 'memory'}, 'indexer_storage': {'args': {}, 'cls': 'memory'}, 'workdir': '/tmp/swh/indexer.fossology.license'}
DEBUG    swh.indexer:indexer.py:161 <swh.indexer.ctags.CtagsIndexer object at 0x7fd6a2b2cd60>: config={'languages': {'c': 'c'}, 'tools': {'configuration': {'command_line': 'ctags --fields=+lnz --sort=no --links=no --output-format=json <filepath>'}, 'name': 'universal-ctags', 'version': '~git7859817b'}, 'storage': {'cls': 'memory'}, 'objstorage': {'args': {}, 'cls': 'memory'}, 'indexer_storage': {'args': {}, 'cls': 'memory'}, 'workdir': '/tmp/swh/indexer.ctags'}
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218902 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
=================================================================================================================================== warnings summary ====================================================================================================================================
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-objstorage/swh/objstorage/factory.py:80: DeprecationWarning: Explicit "args" key is deprecated for objstorage initialization, use class arguments keys directly instead.
    warnings.warn(

swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
swh/web/tests/api/views/test_directory.py::test_api_directory
  /home/anlambert/swh/swh-environment/swh-indexer/swh/indexer/storage/__init__.py:68: DeprecationWarning: Explicit "args" key is deprecated, use keys directly instead.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
-------------------------------------------------------------------------------------------------------------------------------- swh-web custom section ---------------------------------------------------------------------------------------------------------------------------------
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory: Use "pytest --swh-web-random-seed=1634218902 swh/web/tests/api/views/test_directory.py::test_api_directory" to reproduce that test failure with same inputs
================================================================================================================================ short test summary info ================================================================================================================================
FAILED swh/web/tests/api/views/test_directory.py::test_api_directory - assert [{'checksums': {'sha1': '1c34570332c08d4a0ff89ab1d1b7ed5924d7114f', 'sha1_git': '420ebab5ea719a78f9c3059087885fab3eca7...82b58d520ef8a6369'}, 'dir_id': 'cd19126d815470b28919d64b2a8e6a3e37f900dd', 'len...
============================================================================================================================= 1 failed, 6 warnings in 2.75s =============================================================================================================================

Diff Detail

Repository
rDWAPPS Web applications
Branch
tests-random-reproducibility
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24436
Build 38135: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 38134: arc lint + arc unit

Unit TestsFailed

TimeTest
210 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.browse.views.test_directory::Tests / Python tests / test_directory_origin_snapshot_branch_browse
client = <django.test.client.Client object at 0x7f540456ae48> archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7f53fe891cf8> origin_with_multiple_visits = {'archives': ['highlightjs-line-numbers.js.zip', 'highlightjs-line-numbers.js_visit2.zip'], 'id': b'MZ\xc0H_lE[d\xcc\x...s://doi.org/10.5063/schema/codemeta-2.0', 'description': 'Line numbering plugin for Highlight.js'}, 'type': 'git', ...}
223 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.browse.views.test_directory::Tests / Python tests / test_directory_origin_snapshot_revision_browse
client = <django.test.client.Client object at 0x7f53fa737668> archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7f53fa737ba8> origin_with_multiple_visits = {'archives': ['highlightjs-line-numbers.js.zip', 'highlightjs-line-numbers.js_visit2.zip'], 'id': b'MZ\xc0H_lE[d\xcc\x...s://doi.org/10.5063/schema/codemeta-2.0', 'description': 'Line numbering plugin for Highlight.js'}, 'type': 'git', ...}
779 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.random_fixtures_test::Tests / Python tests / test_random_fixture_values
sha1 = '148a71274b4eb55aa377883b61b0d9ce2ec037d2' invalid_sha1 = 'db81a42924691efd5ae6457b4dc86fc20d60efc128f7f51527e60985a33fdbf3bc274074baec0e9dea3ab65b208d62d3257d' sha256 = 'e26175588f6e2cd5037543ce6a057bb215badab564a4ec592f0d1bb1746e7715'
1,115 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.admin.test_origin_save::Tests / Python tests / test_accept_pending_save_request
435 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.admin.test_origin_save::Tests / Python tests / test_add_authorized_origin_url
View Full Test Results (3 Failed · 754 Passed · 5 Skipped)

Event Timeline

Build has FAILED

Patch application report for D6476 (id=23526)

Could not rebase; Attempt merge onto a200f70f2c...

Updating a200f70f..46fc7d7f
Fast-forward
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  14 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 4 files changed, 362 insertions(+), 124 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 46fc7d7fcda3623521937c6ab73d8092ab76c8ff
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitely
      specify the random seed that will be used to run the tests

commit 5fdf48a5def67c2997025b372549c9a7f887f7d9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1178/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1178/console

Build has FAILED

Patch application report for D6476 (id=23526)

Could not rebase; Attempt merge onto a200f70f2c...

Updating a200f70f..46fc7d7f
Fast-forward
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  14 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 4 files changed, 362 insertions(+), 124 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 46fc7d7fcda3623521937c6ab73d8092ab76c8ff
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitely
      specify the random seed that will be used to run the tests

commit 5fdf48a5def67c2997025b372549c9a7f887f7d9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1179/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1179/console

Build has FAILED

Patch application report for D6476 (id=23527)

Could not rebase; Attempt merge onto a200f70f2c...

Updating a200f70f..797e5f82
Fast-forward
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  14 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 4 files changed, 362 insertions(+), 124 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 797e5f826bcac70699e9a3087135c4d72ea51d92
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitly
      specify the random seed that will be used to run the tests

commit 5fdf48a5def67c2997025b372549c9a7f887f7d9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1180/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1180/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 14 2021, 4:09 PM
Harbormaster failed remote builds in B24436: Diff 23527!

Ensure random_fixtures_test.py file is ignored when collecting tests.

Build is green

Patch application report for D6476 (id=23529)

Could not rebase; Attempt merge onto a200f70f2c...

Updating a200f70f..1f5139a3
Fast-forward
 pytest.ini                            |   2 +-
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  14 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 5 files changed, 363 insertions(+), 125 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 1f5139a30836d80d95cdb2682814e66aeaaa6537
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitly
      specify the random seed that will be used to run the tests

commit 5fdf48a5def67c2997025b372549c9a7f887f7d9
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1181/ for more details.

Build is green

Patch application report for D6476 (id=23532)

Could not rebase; Attempt merge onto ed2c2dbed9...

Updating ed2c2dbe..fe1e8a45
Fast-forward
 pytest.ini                            |   2 +-
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  14 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 5 files changed, 363 insertions(+), 125 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit fe1e8a45991f5758a07cee8564885c0f3db69ab3
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitly
      specify the random seed that will be used to run the tests

commit a43a6fa3fbe1d4b18630ce3e03818d37d9ca2322
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1183/ for more details.

ardumont added inline comments.
swh/web/tests/data.py
471

why do you need to sort things out here?

swh/web/tests/data.py
471

Set iterations in Python do not have a guaranteed order across runs so these sorting operations here. Nevertheless swhids is a list so the sorting should not be needed here, need to check.

Remove not needed sorting operation on swhids list.

Build is green

Patch application report for D6476 (id=23540)

Could not rebase; Attempt merge onto 4162a34683...

Merge made by the 'recursive' strategy.
 pytest.ini                            |   2 +-
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  12 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 5 files changed, 362 insertions(+), 124 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 4f60c22ebfb7010cf57eab29149a87c25fe35c8a
Merge: 4162a346 e5125eb4
Author: Jenkins user <jenkins@localhost>
Date:   Thu Oct 14 16:38:37 2021 +0000

    Merge branch 'diff-target' into HEAD

commit e5125eb423b61f25d1e9ed49da3b576fb370eac8
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitly
      specify the random seed that will be used to run the tests

commit a43a6fa3fbe1d4b18630ce3e03818d37d9ca2322
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1185/ for more details.

lgtm

swh/web/tests/data.py
471

right

This revision is now accepted and ready to land.Oct 15 2021, 2:36 PM

Build is green

Patch application report for D6476 (id=23570)

Could not rebase; Attempt merge onto 4162a34683...

Updating 4162a346..61b80b8b
Fast-forward
 pytest.ini                            |   2 +-
 swh/web/tests/conftest.py             | 330 ++++++++++++++++++++++------------
 swh/web/tests/data.py                 |  12 +-
 swh/web/tests/random_fixtures_test.py |  99 ++++++++++
 swh/web/tests/test_random_fixtures.py |  43 +++++
 5 files changed, 362 insertions(+), 124 deletions(-)
 create mode 100644 swh/web/tests/random_fixtures_test.py
 create mode 100644 swh/web/tests/test_random_fixtures.py
Changes applied before test
commit 61b80b8b23146c0c6a017436bd47eca85c9fff77
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 14 15:33:16 2021 +0200

    tests: Ensure failing test with random fixture data can be reproduced
    
    Some tests input data returned by swh-web fixtures are sampled from
    filtered objects stored in a static test archive.
    
    In order to reproduce any failing test with the same fixture data:
    
    - add a new autouse function scope fixture setting the random seed
      to current time before each test execution and putting that seed
      in cache
    
    - implement custom pytest reporting which adds a new section detailing
      the random seeds used for each failing test and the pytests commands
      to reproduce the failures
    
    - add a new pytest option --swh-web-random-seed enabling to explicitly
      specify the random seed that will be used to run the tests

commit efbf0cdc2c4d1ac71879b6924e121d8191047b46
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Oct 13 16:47:55 2021 +0200

    tests/conftest: Put filtered input data in cache
    
    Some tests input data returned by fixtures are sampled from filtered
    objects stored in a static test archive.
    
    In order to avoid repeating the same filtering operations multiple
    times, put these filtered data in cache.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1187/ for more details.