Page MenuHomeSoftware Heritage
Differential D6053

Refactor the use of archive `Storage` object for testing
ClosedPublic
Actions

Authored by aeviso on Aug 3 2021, 4:44 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing
Summary

Remove any use of swh.model.tests.swh_model_data.TEST_OBJECTS, and keep a single fixture for
ArchiveInterface objects based on ArchiveStorage (using swh_storage fixture as parameter).
This avoids duplicating every test for each ArchiveInterface implementation. All
implementations of ArchiveInterface are tested against each other in test_archive_interface.

Also, ArchiveStorage now has a storage attribute to access the underlying StorageInterface
object, so that test no longer needs to receive archive and swh_storage separately. Each
test is responsible for filling archive. storage with the required data specific to the test.

Depends on D6026.

Diff Detail

Event Timeline

aeviso created this revision.Aug 3 2021, 4:44 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 3 2021, 4:44 PM
aeviso added a child revision: D6054: Add test for the different `ProvenanceStorageInterface` implementations.Aug 3 2021, 4:45 PM
swh-public-ci added a comment.Aug 3 2021, 4:54 PM

Build is green

Patch application report for D6053 (id=21905)

Could not rebase; Attempt merge onto 509280132c...

Updating 5092801..70651f4
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb_base.py     |   8 +-
 .../postgresql/provenancedb_with_path.py           |   8 +-
 .../postgresql/provenancedb_without_path.py        |  14 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 200 ++-------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |   4 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 ++++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  19 --
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 154 ++++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   6 +-
 28 files changed, 530 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (70%)
Changes applied before test
commit 70651f45092ae590ee8a66febf4543cb37a0627f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit b738c4610cb84f3f5d10a16f667e4522f7d49377
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/281/ for more details.

Harbormaster completed remote builds in B22871: Diff 21905.Aug 3 2021, 4:54 PM
aeviso requested review of this revision.Aug 3 2021, 4:54 PM
douardda accepted this revision.Aug 5 2021, 12:11 PM
douardda added a subscriber: douardda.

nice job, thx

This revision is now accepted and ready to land.Aug 5 2021, 12:11 PM
aeviso updated this revision to Diff 21961.Aug 6 2021, 3:18 PM

rebase

swh-public-ci added a comment.Aug 6 2021, 3:37 PM

Build is green

Patch application report for D6053 (id=21961)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..d0a122a
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit d0a122af791ed2b07d1278bbc008d5531dff3023
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/287/ for more details.

Harbormaster completed remote builds in B22924: Diff 21961.Aug 6 2021, 3:37 PM
aeviso updated this revision to Diff 21966.Aug 6 2021, 3:49 PM

rebase

swh-public-ci added a comment.Aug 6 2021, 3:56 PM

Build is green

Patch application report for D6053 (id=21966)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..6a207b0
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit 6a207b0e6be02571acea3ca0d2f605d2721b4405
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/289/ for more details.

Harbormaster completed remote builds in B22928: Diff 21966.Aug 6 2021, 3:56 PM
Closed by commit rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing (authored by aeviso). · Explain WhyAug 6 2021, 4:00 PM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
provenance/
3 lines76%
Loading...
postgresql/
6 lines100%
Loading...
tests/
63 lines98%
Loading...
4 lines100%
Loading...
18 lines100%
Loading...
4 lines100%
Loading...
12 lines100%
Loading...
36 lines100%
Loading...
4 lines100%
Loading...
30 lines100%
Loading...
18 lines100%
Loading...
9 lines100%
Loading...

Diff 21966

View Options

swh/provenance/archive.py

Loading...
View Options

swh/provenance/postgresql/archive.py

Loading...
View Options

swh/provenance/tests/conftest.py

Loading...
View Options

swh/provenance/tests/test_archive_interface.py

Loading...
View Options

swh/provenance/tests/test_conftest.py

Loading...
View Options

swh/provenance/tests/test_history_graph.py

Loading...
View Options

swh/provenance/tests/test_isochrone_graph.py

Loading...
View Options

swh/provenance/tests/test_origin_iterator.py

Loading...
View Options

swh/provenance/tests/test_origin_revision_layer.py

Loading...
View Options

swh/provenance/tests/test_provenance_db.py

Loading...
View Options

swh/provenance/tests/test_revision_content_layer.py

Loading...
View Options

swh/provenance/tests/test_revision_iterator.py

Loading...
About · Developer information · Legal