Page MenuHomeSoftware Heritage
Differential D3583

storage_data: Expose contents as content model object
ClosedPublic
Actions

Authored by ardumont on Jul 21 2020, 2:59 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2494: tests: Use data model objects within tests (drop dicts)
Commits
rDSTO98a87fec5fce: storage_data: Expose contents as content model object
Summary

Related to T2494

Depend on D3582

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jul 21 2020, 2:59 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 21 2020, 2:59 PM
swh-public-ci added a comment.Jul 21 2020, 3:06 PM

Build has FAILED

Patch application report for D3583 (id=12607)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..3a14a04c
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            |  37 +++---
 swh/storage/storage.py                  |  25 +---
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 146 ++++++++++------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +--
 swh/storage/tests/test_pytest_plugin.py |  27 ++--
 swh/storage/tests/test_retry.py         | 113 ++++++-----------
 swh/storage/tests/test_storage.py       | 210 ++++++--------------------------
 swh/storage/validate.py                 | 148 ----------------------
 13 files changed, 187 insertions(+), 561 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit 3a14a04cd011bd3a921d211009c911ee425c5a93
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit 5d88eadbdbb166851d1bbb5e5d9c44b418caabde
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/521/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/521/console

Harbormaster failed remote builds in B13833: Diff 12607!Jul 21 2020, 3:06 PM
ardumont updated this revision to Diff 12608.Jul 21 2020, 3:08 PM

Fix missing content_no_data keys to remove (test_retry was not happy)

swh-public-ci added a comment.Jul 21 2020, 3:15 PM

Build is green

Patch application report for D3583 (id=12608)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..ba3af6ff
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            |  37 +++---
 swh/storage/storage.py                  |  25 +---
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 146 ++++++++++------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +--
 swh/storage/tests/test_pytest_plugin.py |  27 ++--
 swh/storage/tests/test_retry.py         | 136 ++++++++-------------
 swh/storage/tests/test_storage.py       | 210 ++++++--------------------------
 swh/storage/validate.py                 | 148 ----------------------
 13 files changed, 201 insertions(+), 570 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit ba3af6ffa86ce77df2d92d926b7bafffc1fdc91d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit 5d88eadbdbb166851d1bbb5e5d9c44b418caabde
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/522/ for more details.

Harbormaster completed remote builds in B13834: Diff 12608.Jul 21 2020, 3:15 PM
ardumont updated this revision to Diff 12610.Jul 21 2020, 3:20 PM

Fix missing "," typo

swh-public-ci added a comment.Jul 21 2020, 3:29 PM

Build is green

Patch application report for D3583 (id=12610)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..ab48fec4
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            |  37 +++---
 swh/storage/storage.py                  |  25 +---
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 146 ++++++++++------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +--
 swh/storage/tests/test_pytest_plugin.py |  28 +++--
 swh/storage/tests/test_retry.py         | 136 ++++++++-------------
 swh/storage/tests/test_storage.py       | 210 ++++++--------------------------
 swh/storage/validate.py                 | 148 ----------------------
 13 files changed, 202 insertions(+), 570 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit ab48fec43eb1e8a07dce34153667f6a220605464
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit 5d88eadbdbb166851d1bbb5e5d9c44b418caabde
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/524/ for more details.

Harbormaster completed remote builds in B13836: Diff 12610.Jul 21 2020, 3:29 PM
ardumont updated this revision to Diff 12614.Jul 21 2020, 4:24 PM

Rebase

swh-public-ci added a comment.Jul 21 2020, 4:47 PM

Build is green

Patch application report for D3583 (id=12614)

Could not rebase; Attempt merge onto e0152b0407...

Updating e0152b04..98a87fec
Fast-forward
 swh/storage/__init__.py                 |   3 -
 swh/storage/cassandra/storage.py        |   2 +
 swh/storage/pytest_plugin.py            |  37 +++---
 swh/storage/storage.py                  |  25 +---
 swh/storage/tests/conftest.py           |   6 -
 swh/storage/tests/storage_data.py       | 146 ++++++++++------------
 swh/storage/tests/test_api_client.py    |   5 +-
 swh/storage/tests/test_buffer.py        |   8 +-
 swh/storage/tests/test_cassandra.py     |  18 +--
 swh/storage/tests/test_pytest_plugin.py |  27 ++--
 swh/storage/tests/test_retry.py         | 136 ++++++++-------------
 swh/storage/tests/test_storage.py       | 210 ++++++--------------------------
 swh/storage/validate.py                 | 148 ----------------------
 13 files changed, 202 insertions(+), 569 deletions(-)
 delete mode 100644 swh/storage/validate.py
Changes applied before test
commit 98a87fec5fce34d2051d453db834a5318cf40605
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:59:09 2020 +0200

    storage_data: Expose contents as content model object
    
    Related to T2494

commit a23b748995a06387511b7a386febd380dc303bf0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 14:43:21 2020 +0200

    pytest_plugin: Drop unnecessary back and forth conversion
    
    This is preparatory work to incrementally migrate the sample_data fixture to
    use model objects directly.
    
    Related to T2494

commit 6338ad2769f452bd17f7aff7275adec696acb842
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Jul 21 09:42:14 2020 +0200

    Drop validate proxy
    
    The validate proxy was initially an helper to ease the transition from the use
    of dicts towards model objects in "*_add" production endpoints. It was not
    removed immediately and grew some behavior it should not have (notably revision
    conversion so the comparison within those related tests work).
    
    After finally migrated away from dicts within the tests, we can now drop it [1].
    
    Note that this moves the extra revision conversion behavior from the validate
    proxy to those related tests. This extra step will also disappear when we
    finally move the "*_get" endpoints to return model objects as well.
    
    Note:
    - This drops fixture redefinitions in the process (introduced so we could have
    that validate proxy at the time).
    - Remove the "validate" keyword from the get_storage function (so no longer
    possible to instantiate one [2])
    
    [1] T2994
    
    [2] which, practically, is the case today, nothing runs on production with it.
    
    Related to T2499

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/528/ for more details.

Harbormaster completed remote builds in B13840: Diff 12614.Jul 21 2020, 4:47 PM
ardumont mentioned this in D3584: storage_data: Remove unused fixture data.Jul 21 2020, 9:14 PM
ardumont edited the summary of this revision. (Show Details)
vlorentz accepted this revision.Jul 22 2020, 10:21 AM
This revision is now accepted and ready to land.Jul 22 2020, 10:21 AM
Closed by commit rDSTO98a87fec5fce: storage_data: Expose contents as content model object (authored by ardumont). · Explain WhyJul 22 2020, 11:00 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTO98a87fec5fce: storage_data: Expose contents as content model object.

Revision Contents
Changeset List

PathSize
swh/
storage/
6 lines
tests/
146 lines
8 lines
4 lines
23 lines
27 lines

Diff 12632

View Options

swh/storage/pytest_plugin.py

Loading...
View Options

swh/storage/tests/storage_data.py

Loading...
View Options

swh/storage/tests/test_buffer.py

Loading...
View Options

swh/storage/tests/test_pytest_plugin.py

Loading...
View Options

swh/storage/tests/test_retry.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal