Paths

Table of Contentst

Differential D3591

pytest_plugin: Drop sample_data to in favor of sample_data_model
ClosedPublic
Actions

Authored by ardumont on Jul 21 2020, 6:09 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T2494: tests: Use data model objects within tests (drop dicts)

Commits

rDSTOe005900bd7ed: pytest_plugin: Drop sample_data in favor of sample_data_model

Summary

I'll rename sample_data_model to sample_data in a second diff (so that diff
remains reviewable enough).

Note:
The data are now tuple instead of lists

Related to T2494

Depend on D3590

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 13851
Build 21238: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 21237: arc lint + arc unit

Event Timeline

ardumont created this revision.Jul 21 2020, 6:09 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 21 2020, 6:09 PM

Build is green

Patch application report for D3591 (id=12623)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..722196db
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            | 101 +---
 swh/storage/storage.py                  |  25 +-
 swh/storage/tests/algos/test_origin.py  |  21 +-
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 813 +++++++++++++++-----------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +-
 swh/storage/tests/test_pytest_plugin.py |  47 +-
 swh/storage/tests/test_retry.py         | 138 ++----
 swh/storage/tests/test_storage.py       | 291 +++---------
 swh/storage/validate.py                 | 148 ------
 14 files changed, 563 insertions(+), 1063 deletions(-)
 delete mode 100644 swh/storage/validate.py

Changes applied before test

commit 722196dbd7d7a2079bd8f4cc689901bb97e0aea1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data to the benefits of sample_data_model
    
    Related to T2494

commit bbe840e089d18148a9c5b38e8ac71d4106bc3582
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:43:22 2020 +0200

    storage_data: Expose snapshots as model objects
    
    Related to T2494

commit d0cf317e7c8705e943ee9ea8f144e38803ecd7df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:32:03 2020 +0200

    storage_data: Expose release as model objects
    
    Related to T2494

commit 3be5327f0a0ee104e99771736828173150ced724
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:21:20 2020 +0200

    storage_data: Expose origin_visits as model objects
    
    Related to T2494

commit bcc0aee8d9cb7bdd4790c2155947e66d3bbc06a2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:52:22 2020 +0200

    storage_data: Expose origins as model objects
    
    Related to T2494

commit d4cd33c3aab44d2f3c17bccebc2b756e499aa18b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:24:16 2020 +0200

    storage_data: Expose revisions as model objects
    
    Related to T2494

commit 955b6e28533c48caaa18a38e52ae2d80b5033c6b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:35:23 2020 +0200

    storage_data: Expose directories as directory model objects
    
    Related to T2494

commit 95dbdf792ea279f73c76b286658a7ef98accb9d3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:04:04 2020 +0200

    storage_data: Remove unused fixture data
    
    Less to maintain
    
    Related to T2494

commit 98a87fec5fce34d2051d453db834a5318cf40605
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit a23b748995a06387511b7a386febd380dc303bf0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/537/ for more details.

Harbormaster completed remote builds in B13849: Diff 12623.Jul 21 2020, 6:39 PM

ardumont added a child revision: D3592: pytest_plugin: Rename sample_data_model to sample_data.Jul 21 2020, 9:12 PM

ardumont edited the summary of this revision. (Show Details)

Fix metadata_authorities and fetchers to tuple (like the rest of the module).
(so that's immutable)

Build is green

Patch application report for D3591 (id=12625)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..46bcd09f
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            | 101 +---
 swh/storage/storage.py                  |  25 +-
 swh/storage/tests/algos/test_origin.py  |  21 +-
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 813 +++++++++++++++-----------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +-
 swh/storage/tests/test_pytest_plugin.py |  47 +-
 swh/storage/tests/test_retry.py         | 138 ++----
 swh/storage/tests/test_storage.py       | 291 +++---------
 swh/storage/validate.py                 | 148 ------
 14 files changed, 563 insertions(+), 1063 deletions(-)
 delete mode 100644 swh/storage/validate.py

Changes applied before test

commit 46bcd09f2db433883170dff678080a22c03b94d1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data to the benefits of sample_data_model
    
    Related to T2494

commit bbe840e089d18148a9c5b38e8ac71d4106bc3582
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:43:22 2020 +0200

    storage_data: Expose snapshots as model objects
    
    Related to T2494

commit d0cf317e7c8705e943ee9ea8f144e38803ecd7df
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:32:03 2020 +0200

    storage_data: Expose release as model objects
    
    Related to T2494

commit 3be5327f0a0ee104e99771736828173150ced724
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 17:21:20 2020 +0200

    storage_data: Expose origin_visits as model objects
    
    Related to T2494

commit bcc0aee8d9cb7bdd4790c2155947e66d3bbc06a2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:52:22 2020 +0200

    storage_data: Expose origins as model objects
    
    Related to T2494

commit d4cd33c3aab44d2f3c17bccebc2b756e499aa18b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 16:24:16 2020 +0200

    storage_data: Expose revisions as model objects
    
    Related to T2494

commit 955b6e28533c48caaa18a38e52ae2d80b5033c6b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:35:23 2020 +0200

    storage_data: Expose directories as directory model objects
    
    Related to T2494

commit 95dbdf792ea279f73c76b286658a7ef98accb9d3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 15:04:04 2020 +0200

    storage_data: Remove unused fixture data
    
    Less to maintain
    
    Related to T2494

commit 98a87fec5fce34d2051d453db834a5318cf40605
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit a23b748995a06387511b7a386febd380dc303bf0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/539/ for more details.

Harbormaster completed remote builds in B13851: Diff 12625.Jul 22 2020, 9:02 AM

s/to the benefit of/in favor of/

swh/storage/pytest_plugin.py
191	`Tuple[BaseModel]` is the type of tuples of length 1; use `Tuple[BaseModel, ...]` instead.

This revision now requires changes to proceed.Jul 22 2020, 10:31 AM

ardumont retitled this revision from pytest_plugin: Drop sample_data to the benefit of sample_data_model to pytest_plugin: Drop sample_data to in favor of sample_data_model.Jul 22 2020, 10:46 AM

ardumont added inline comments.

swh/storage/pytest_plugin.py
191	Did not realize that, thanks for the heads up.

Rework commit message
Adapt mistyped fixture

Build is green

Patch application report for D3591 (id=12640)

Rebasing onto bbe840e089...

Current branch diff-target is up to date.

Changes applied before test

commit e005900bd7edd3dd12b10beb2cf9c00db8488093
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 18:07:44 2020 +0200

    pytest_plugin: Drop sample_data in favor of sample_data_model
    
    Related to T2494

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/543/ for more details.

Harbormaster completed remote builds in B13865: Diff 12640.Jul 22 2020, 11:47 AM

vlorentz accepted this revision.Jul 22 2020, 4:52 PM

This revision is now accepted and ready to land.Jul 22 2020, 4:52 PM

Closed by commit rDSTOe005900bd7ed: pytest_plugin: Drop sample_data in favor of sample_data_model (authored by ardumont). · Explain WhyJul 22 2020, 4:57 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSTOe005900bd7ed: pytest_plugin: Drop sample_data in favor of sample_data_model.