Page MenuHomeSoftware Heritage
Differential D3593

pytest_plugin: Make sample_data an object with attributes instead of dict
ClosedPublic
Actions

Authored by ardumont on Jul 22 2020, 8:57 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2494: tests: Use data model objects within tests (drop dicts)
Commits
rDSTOccbd2e9c21e6: pytest_plugin: Make sample_data an object
Summary

Note that this:

  • drops the no longer needed copy done by the StorageData instance (used by sample_data) since now it returned immutable BaseModel objects.
  • centralizes some left-over tests to use sample_data as well

This concludes the tests refactoring about model objects. Now all objects used
within tests are immutable BaseModel objects.

Furthermore, the storage exposes the sample_data fixture so it can be reused by
client parts without having to redefine their own (see [1] as a potential
refactoring)

[1] https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/tests/test_init.py$46-164

Closes T2494

Depends on D3592

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13853
Build 21242: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 21241: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_origin::test_origin_get_latest_visit_status_filter_snapshot
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f5a185095f8> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f5b2e64afd0>
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_origin::test_origin_get_latest_visit_status_filter_status
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f5a182fe4e0> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f5b2e64afd0>
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_origin::test_origin_get_latest_visit_status_filter_type
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f5a78213a58> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f5b2e64afd0>
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_origin::test_origin_get_latest_visit_status_none
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f5a18253da0> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f5b2e64afd0>
623 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_snapshot::test_snapshot_get_latest
swh_storage = <swh.storage.in_memory.InMemoryStorage object at 0x7f5a187cec18> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f5b2e64afd0>
View Full Test Results (10 Failed · 745 Passed · 17 Skipped)

Event Timeline

ardumont created this revision.Jul 22 2020, 8:57 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 22 2020, 8:57 AM
ardumont edited the summary of this revision. (Show Details)Jul 22 2020, 9:01 AM
swh-public-ci added a comment.Jul 22 2020, 9:22 AM

Build has FAILED

Patch application report for D3593 (id=12628)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..bd96d5c8
Fast-forward
 swh/storage/__init__.py                      |    3 -
 swh/storage/cassandra/storage.py             |    2 +
 swh/storage/pytest_plugin.py                 |   96 +--
 swh/storage/storage.py                       |   25 +-
 swh/storage/tests/algos/test_origin.py       |   72 +-
 swh/storage/tests/algos/test_snapshot.py     |   16 +-
 swh/storage/tests/conftest.py                |    6 -
 swh/storage/tests/storage_data.py            |  819 +++++++++----------
 swh/storage/tests/test_api_client.py         |    9 +-
 swh/storage/tests/test_buffer.py             |   78 +-
 swh/storage/tests/test_cassandra.py          |   38 +-
 swh/storage/tests/test_filter.py             |   20 +-
 swh/storage/tests/test_pytest_plugin.py      |   89 +--
 swh/storage/tests/test_retry.py              |  250 +++---
 swh/storage/tests/test_revision_bw_compat.py |    6 +-
 swh/storage/tests/test_storage.py            | 1099 ++++++++++++--------------
 swh/storage/validate.py                      |  148 ----
 17 files changed, 1154 insertions(+), 1622 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit bd96d5c8fecfff278b3edb652785004cf8b19e95
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 22 08:46:10 2020 +0200

    pytest_plugin: Make sample_data an object
    
    Note that this:
    - drops the no longer needed copy done by the StorageData instance (used by
    sample_data) since now it returned immutable BaseModel objects.
    
    - centralizes some left-over tests to use sample_data as well

commit 67f261ead524299097c1993bfcebf71a9157077d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:15:41 2020 +0200

    pytest_plugin: Rename sample_data_model to sample_data
    
    Related to T2494

commit 46bcd09f2db433883170dff678080a22c03b94d1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data to the benefits of sample_data_model
    
    Related to T2494

commit bbe840e089d18148a9c5b38e8ac71d4106bc3582
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:43:22 2020 +0200

    storage_data: Expose snapshots as model objects
    
    Related to T2494

commit d0cf317e7c8705e943ee9ea8f144e38803ecd7df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:32:03 2020 +0200

    storage_data: Expose release as model objects
    
    Related to T2494

commit 3be5327f0a0ee104e99771736828173150ced724
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:21:20 2020 +0200

    storage_data: Expose origin_visits as model objects
    
    Related to T2494

commit bcc0aee8d9cb7bdd4790c2155947e66d3bbc06a2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:52:22 2020 +0200

    storage_data: Expose origins as model objects
    
    Related to T2494

commit d4cd33c3aab44d2f3c17bccebc2b756e499aa18b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:24:16 2020 +0200

    storage_data: Expose revisions as model objects
    
    Related to T2494

commit 955b6e28533c48caaa18a38e52ae2d80b5033c6b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:35:23 2020 +0200

    storage_data: Expose directories as directory model objects
    
    Related to T2494

commit 95dbdf792ea279f73c76b286658a7ef98accb9d3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:04:04 2020 +0200

    storage_data: Remove unused fixture data
    
    Less to maintain
    
    Related to T2494

commit 98a87fec5fce34d2051d453db834a5318cf40605
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit a23b748995a06387511b7a386febd380dc303bf0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/541/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/541/console

Harbormaster failed remote builds in B13853: Diff 12628!Jul 22 2020, 9:23 AM
ardumont edited the summary of this revision. (Show Details)Jul 22 2020, 9:26 AM
ardumont added a parent revision: D2592: webapp0: Switch to cassandra storage.
ardumont edited the summary of this revision. (Show Details)
ardumont added a parent revision: D3592: pytest_plugin: Rename sample_data_model to sample_data.
ardumont updated this revision to Diff 12629.Jul 22 2020, 9:43 AM
  • Fix missing references update (build should go green)
  • Simplify sample_data declaration calls when using a simple object
ardumont removed a parent revision: D2592: webapp0: Switch to cassandra storage.Jul 22 2020, 9:47 AM
swh-public-ci added a comment.Jul 22 2020, 9:55 AM

Build is green

Patch application report for D3593 (id=12629)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..7e08b878
Fast-forward
 swh/storage/__init__.py                      |    3 -
 swh/storage/cassandra/storage.py             |    2 +
 swh/storage/pytest_plugin.py                 |   96 +--
 swh/storage/storage.py                       |   25 +-
 swh/storage/tests/algos/test_origin.py       |   72 +-
 swh/storage/tests/algos/test_snapshot.py     |   16 +-
 swh/storage/tests/conftest.py                |    6 -
 swh/storage/tests/storage_data.py            |  819 +++++++++----------
 swh/storage/tests/test_api_client.py         |    9 +-
 swh/storage/tests/test_buffer.py             |  104 ++-
 swh/storage/tests/test_cassandra.py          |   38 +-
 swh/storage/tests/test_filter.py             |   20 +-
 swh/storage/tests/test_pytest_plugin.py      |   89 +--
 swh/storage/tests/test_retry.py              |  251 +++---
 swh/storage/tests/test_revision_bw_compat.py |    6 +-
 swh/storage/tests/test_storage.py            | 1099 ++++++++++++--------------
 swh/storage/validate.py                      |  148 ----
 17 files changed, 1166 insertions(+), 1637 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit 7e08b87829f480f9ad5c1ef574b9963c96dc58eb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 22 08:46:10 2020 +0200

    pytest_plugin: Make sample_data an object
    
    Note that this:
    - drops the no longer needed copy done by the StorageData instance (used by
    sample_data) since now it returned immutable BaseModel objects.
    
    - centralizes some left-over tests to use sample_data as well

commit 67f261ead524299097c1993bfcebf71a9157077d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:15:41 2020 +0200

    pytest_plugin: Rename sample_data_model to sample_data
    
    Related to T2494

commit 46bcd09f2db433883170dff678080a22c03b94d1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data to the benefits of sample_data_model
    
    Related to T2494

commit bbe840e089d18148a9c5b38e8ac71d4106bc3582
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:43:22 2020 +0200

    storage_data: Expose snapshots as model objects
    
    Related to T2494

commit d0cf317e7c8705e943ee9ea8f144e38803ecd7df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:32:03 2020 +0200

    storage_data: Expose release as model objects
    
    Related to T2494

commit 3be5327f0a0ee104e99771736828173150ced724
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:21:20 2020 +0200

    storage_data: Expose origin_visits as model objects
    
    Related to T2494

commit bcc0aee8d9cb7bdd4790c2155947e66d3bbc06a2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:52:22 2020 +0200

    storage_data: Expose origins as model objects
    
    Related to T2494

commit d4cd33c3aab44d2f3c17bccebc2b756e499aa18b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:24:16 2020 +0200

    storage_data: Expose revisions as model objects
    
    Related to T2494

commit 955b6e28533c48caaa18a38e52ae2d80b5033c6b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:35:23 2020 +0200

    storage_data: Expose directories as directory model objects
    
    Related to T2494

commit 95dbdf792ea279f73c76b286658a7ef98accb9d3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:04:04 2020 +0200

    storage_data: Remove unused fixture data
    
    Less to maintain
    
    Related to T2494

commit 98a87fec5fce34d2051d453db834a5318cf40605
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit a23b748995a06387511b7a386febd380dc303bf0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/542/ for more details.

Harbormaster completed remote builds in B13854: Diff 12629.Jul 22 2020, 9:55 AM
vlorentz requested changes to this revision.Jul 22 2020, 10:34 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/tests/storage_data.py
38–48

the point of using object attributes instead of a dict was to allow type checking; but the dynamic access make it impossible. Could you write the set of attributes explicitly here?

This revision now requires changes to proceed.Jul 22 2020, 10:34 AM
ardumont added inline comments.Jul 22 2020, 10:52 AM
swh/storage/tests/storage_data.py
38–48

Yeah, i went the easy way because the test_pytest_plugin checked that already.
But yeah, that's kinda indirect...

I'll adapt accordingly.

Thanks ;)

ardumont updated this revision to Diff 12642.EditedJul 22 2020, 12:21 PM
  • Adapt according to review
  • Fix remaining tests not using sample_data
  • moved tests from test_pytest_plugin to test_storage_data (as it was really about testing the storage data)
vlorentz added inline comments.Jul 22 2020, 12:28 PM
swh/storage/tests/storage_data.py
290–291

add an underscore prefix

453–455

prefix

swh-public-ci added a comment.Jul 22 2020, 12:34 PM

Build is green

Patch application report for D3593 (id=12642)

Could not rebase; Attempt merge onto bbe840e089...

Updating bbe840e0..adf7c69f
Fast-forward
 swh/storage/pytest_plugin.py                 |   91 +--
 swh/storage/tests/algos/test_origin.py       |   69 +-
 swh/storage/tests/algos/test_snapshot.py     |   16 +-
 swh/storage/tests/storage_data.py            | 1013 +++++++++++++-------------
 swh/storage/tests/test_api_client.py         |    4 +-
 swh/storage/tests/test_buffer.py             |  102 ++-
 swh/storage/tests/test_cassandra.py          |   20 +-
 swh/storage/tests/test_filter.py             |   20 +-
 swh/storage/tests/test_pytest_plugin.py      |   55 +-
 swh/storage/tests/test_retry.py              |  164 ++---
 swh/storage/tests/test_revision_bw_compat.py |    6 +-
 swh/storage/tests/test_storage.py            |  856 +++++++++++-----------
 swh/storage/tests/test_storage_data.py       |   29 +
 13 files changed, 1192 insertions(+), 1253 deletions(-)
 create mode 100644 swh/storage/tests/test_storage_data.py
Changes applied before test
commit adf7c69fc07f95ac868989f732820f39e2cf1a92
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 22 08:46:10 2020 +0200

    pytest_plugin: Make sample_data an object
    
    Note that this:
    - drops the no longer needed copy done by the StorageData instance (used by
    sample_data) since now it returned immutable BaseModel objects.
    
    - centralizes some left-over tests to use sample_data as well

commit 67a909e0fb69be9ea11c951d771002f37544d847
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:15:41 2020 +0200

    pytest_plugin: Rename sample_data_model to sample_data
    
    Related to T2494

commit e005900bd7edd3dd12b10beb2cf9c00db8488093
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data in favor of sample_data_model
    
    Related to T2494

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/545/ for more details.

Harbormaster completed remote builds in B13867: Diff 12642.Jul 22 2020, 12:34 PM
ardumont added inline comments.Jul 22 2020, 12:37 PM
swh/storage/tests/storage_data.py
290–291

It's not even used...

453–455

?

Those are used directly by some tests so no _ prefix (if that's what you meant ;)

ardumont updated this revision to Diff 12643.Jul 22 2020, 12:40 PM

Drop unused data tests

swh-public-ci added a comment.Jul 22 2020, 12:53 PM

Build is green

Patch application report for D3593 (id=12643)

Could not rebase; Attempt merge onto bbe840e089...

Updating bbe840e0..ccbd2e9c
Fast-forward
 swh/storage/pytest_plugin.py                 |   91 +--
 swh/storage/tests/algos/test_origin.py       |   69 +-
 swh/storage/tests/algos/test_snapshot.py     |   16 +-
 swh/storage/tests/storage_data.py            | 1010 +++++++++++++-------------
 swh/storage/tests/test_api_client.py         |    4 +-
 swh/storage/tests/test_buffer.py             |  102 ++-
 swh/storage/tests/test_cassandra.py          |   20 +-
 swh/storage/tests/test_filter.py             |   20 +-
 swh/storage/tests/test_pytest_plugin.py      |   55 +-
 swh/storage/tests/test_retry.py              |  164 +++--
 swh/storage/tests/test_revision_bw_compat.py |    6 +-
 swh/storage/tests/test_storage.py            |  856 +++++++++++-----------
 swh/storage/tests/test_storage_data.py       |   29 +
 13 files changed, 1189 insertions(+), 1253 deletions(-)
 create mode 100644 swh/storage/tests/test_storage_data.py
Changes applied before test
commit ccbd2e9c21e6862fe8efbbf9b99e4ac3b838be97
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 22 08:46:10 2020 +0200

    pytest_plugin: Make sample_data an object
    
    Note that this:
    - drops the no longer needed copy done by the StorageData instance (used by
    sample_data) since now it returned immutable BaseModel objects.
    
    - centralizes some left-over tests to use sample_data as well

commit 67a909e0fb69be9ea11c951d771002f37544d847
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:15:41 2020 +0200

    pytest_plugin: Rename sample_data_model to sample_data
    
    Related to T2494

commit e005900bd7edd3dd12b10beb2cf9c00db8488093
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data in favor of sample_data_model
    
    Related to T2494

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/546/ for more details.

Harbormaster completed remote builds in B13868: Diff 12643.Jul 22 2020, 12:53 PM
vlorentz accepted this revision.Jul 22 2020, 4:52 PM
vlorentz added inline comments.
swh/storage/tests/storage_data.py
453–455

ok

This revision is now accepted and ready to land.Jul 22 2020, 4:52 PM
Closed by commit rDSTOccbd2e9c21e6: pytest_plugin: Make sample_data an object (authored by ardumont). · Explain WhyJul 22 2020, 4:57 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOccbd2e9c21e6: pytest_plugin: Make sample_data an object.

Revision Contents
Changeset List

PathSize
swh/
storage/
27 lines
tests/
algos/
45 lines
6 lines
42 lines
10 lines
38 lines
80 lines
575 lines

Diff 12628

View Options

swh/storage/pytest_plugin.py

Loading...
View Options

swh/storage/tests/algos/test_origin.py

Loading...
View Options

swh/storage/tests/storage_data.py

Loading...
View Options

swh/storage/tests/test_buffer.py

Loading...
View Options

swh/storage/tests/test_filter.py

Loading...
View Options

swh/storage/tests/test_pytest_plugin.py

Loading...
View Options

swh/storage/tests/test_retry.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal