Page MenuHomeSoftware Heritage

Refactor the use of archive `Storage` object for testing
ClosedPublic

Authored by aeviso on Aug 3 2021, 4:44 PM.

Details

Summary

Remove any use of swh.model.tests.swh_model_data.TEST_OBJECTS, and keep a single fixture for
ArchiveInterface objects based on ArchiveStorage (using swh_storage fixture as parameter).
This avoids duplicating every test for each ArchiveInterface implementation. All
implementations of ArchiveInterface are tested against each other in test_archive_interface.

Also, ArchiveStorage now has a storage attribute to access the underlying StorageInterface
object, so that test no longer needs to receive archive and swh_storage separately. Each
test is responsible for filling archive. storage with the required data specific to the test.

Depends on D6026.

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6053 (id=21905)

Could not rebase; Attempt merge onto 509280132c...

Updating 5092801..70651f4
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb_base.py     |   8 +-
 .../postgresql/provenancedb_with_path.py           |   8 +-
 .../postgresql/provenancedb_without_path.py        |  14 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 200 ++-------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |   4 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 ++++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  19 --
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 154 ++++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   6 +-
 28 files changed, 530 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (70%)
Changes applied before test
commit 70651f45092ae590ee8a66febf4543cb37a0627f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit b738c4610cb84f3f5d10a16f667e4522f7d49377
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/281/ for more details.

aeviso requested review of this revision.Aug 3 2021, 4:54 PM
douardda added a subscriber: douardda.

nice job, thx

This revision is now accepted and ready to land.Aug 5 2021, 12:11 PM

Build is green

Patch application report for D6053 (id=21961)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..d0a122a
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit d0a122af791ed2b07d1278bbc008d5531dff3023
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/287/ for more details.

Build is green

Patch application report for D6053 (id=21966)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..6a207b0
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit 6a207b0e6be02571acea3ca0d2f605d2721b4405
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/289/ for more details.