Page MenuHomeSoftware Heritage
Differential D6053

Refactor the use of archive `Storage` object for testing
ClosedPublic
Actions

Authored by aeviso on Aug 3 2021, 4:44 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing
Summary

Remove any use of swh.model.tests.swh_model_data.TEST_OBJECTS, and keep a single fixture for
ArchiveInterface objects based on ArchiveStorage (using swh_storage fixture as parameter).
This avoids duplicating every test for each ArchiveInterface implementation. All
implementations of ArchiveInterface are tested against each other in test_archive_interface.

Also, ArchiveStorage now has a storage attribute to access the underlying StorageInterface
object, so that test no longer needs to receive archive and swh_storage separately. Each
test is responsible for filling archive. storage with the required data specific to the test.

Depends on D6026.

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

aeviso created this revision.Aug 3 2021, 4:44 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 3 2021, 4:44 PM
aeviso added a child revision: D6054: Add test for the different `ProvenanceStorageInterface` implementations.Aug 3 2021, 4:45 PM
swh-public-ci added a comment.Aug 3 2021, 4:54 PM

Build is green

Patch application report for D6053 (id=21905)

Could not rebase; Attempt merge onto 509280132c...

Updating 5092801..70651f4
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb_base.py     |   8 +-
 .../postgresql/provenancedb_with_path.py           |   8 +-
 .../postgresql/provenancedb_without_path.py        |  14 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 200 ++-------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |   4 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 ++++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  19 --
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 154 ++++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   6 +-
 28 files changed, 530 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (70%)
Changes applied before test
commit 70651f45092ae590ee8a66febf4543cb37a0627f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit b738c4610cb84f3f5d10a16f667e4522f7d49377
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/281/ for more details.

Harbormaster completed remote builds in B22871: Diff 21905.Aug 3 2021, 4:54 PM
aeviso requested review of this revision.Aug 3 2021, 4:54 PM
douardda accepted this revision.Aug 5 2021, 12:11 PM
douardda added a subscriber: douardda.

nice job, thx

This revision is now accepted and ready to land.Aug 5 2021, 12:11 PM
aeviso updated this revision to Diff 21961.Aug 6 2021, 3:18 PM

rebase

swh-public-ci added a comment.Aug 6 2021, 3:37 PM

Build is green

Patch application report for D6053 (id=21961)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..d0a122a
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit d0a122af791ed2b07d1278bbc008d5531dff3023
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/287/ for more details.

Harbormaster completed remote builds in B22924: Diff 21961.Aug 6 2021, 3:37 PM
aeviso updated this revision to Diff 21966.Aug 6 2021, 3:49 PM

rebase

swh-public-ci added a comment.Aug 6 2021, 3:56 PM

Build is green

Patch application report for D6053 (id=21966)

Could not rebase; Attempt merge onto 3b145f15c2...

Updating 3b145f1..6a207b0
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +-
 swh/provenance/api/serializers.py                  |  11 +-
 swh/provenance/api/server.py                       |  35 ++--
 swh/provenance/archive.py                          |   3 +
 swh/provenance/cli.py                              |   2 -
 swh/provenance/postgresql/archive.py               |   8 +-
 swh/provenance/postgresql/provenancedb.py          |  12 +-
 swh/provenance/storage/archive.py                  |   2 +-
 swh/provenance/tests/conftest.py                   | 205 +++------------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   9 +-
 .../tests/data/generate_storage_from_git.py        |  34 ++--
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_archive_interface.py     |   4 +-
 swh/provenance/tests/test_cli.py                   |   5 +-
 swh/provenance/tests/test_conftest.py              |  18 +-
 swh/provenance/tests/test_history_graph.py         |   4 +-
 swh/provenance/tests/test_isochrone_graph.py       |  12 +-
 swh/provenance/tests/test_origin_iterator.py       |  36 ++--
 swh/provenance/tests/test_origin_revision_layer.py | 190 +++++++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  30 ---
 swh/provenance/tests/test_provenance_storage.py    |   2 +-
 ...euristics.py => test_revision_content_layer.py} | 150 +++++++++++++--
 swh/provenance/tests/test_revision_iterator.py     |   9 +-
 26 files changed, 535 insertions(+), 319 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit 6a207b0e6be02571acea3ca0d2f605d2721b4405
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 15:37:43 2021 +0200

    Refactor the use of archive `Storage` object for testing
    
    Remove any use of `swh.model.tests.swh_model_data.TEST_OBJECTS`, and keep a single fixture for
    `ArchiveInterface` objects based on `ArchiveStorage` (using `swh_storage` fixture as parameter).
    This avoids duplicating every test for each `ArchiveInterface` implementation. All
    implementations of `ArchiveInterface` are tested against each other in `test_archive_interface`.
    
    Also, `ArchiveStorage` now has a `storage` attribute to access the underlying `StorageInterface`
    object, so that test no longer needs to receive `archive` and `swh_storage` separately. Each
    test is responsible for filling `archive. storage` with the required data specific to the test.

commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/289/ for more details.

Harbormaster completed remote builds in B22928: Diff 21966.Aug 6 2021, 3:56 PM
Closed by commit rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing (authored by aeviso). · Explain WhyAug 6 2021, 4:00 PM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV6a207b0e6be0: Refactor the use of archive `Storage` object for testing.

Revision Contents
Changeset List

Diff 21972

View Options

swh/provenance/archive.py

Loading...
View Options

swh/provenance/postgresql/archive.py

Loading...
View Options

swh/provenance/tests/conftest.py

Loading...
View Options

swh/provenance/tests/test_archive_interface.py

Loading...
View Options

swh/provenance/tests/test_conftest.py

Loading...
View Options

swh/provenance/tests/test_history_graph.py

Loading...
View Options

swh/provenance/tests/test_isochrone_graph.py

Loading...
View Options

swh/provenance/tests/test_origin_iterator.py

Loading...
View Options

swh/provenance/tests/test_origin_revision_layer.py

Loading...
View Options

swh/provenance/tests/test_provenance_db.py

Loading...
View Options

swh/provenance/tests/test_revision_content_layer.py

Loading...
View Options

swh/provenance/tests/test_revision_iterator.py

Loading...
About · Developer information · Legal