Page MenuHomeSoftware Heritage
Differential D3733

storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
ClosedPublic
Actions

Authored by ardumont on Aug 7 2020, 12:45 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T645: Type swh-storage endpoints with swh.model objects
Commits
rDSTOd9ff3912d5ab: storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
Summary

Impacts:

$ grep -r "content_get_metadata(" $SWH_ENVIRONMENT_HOME/**/swh/* | grep -v swh-storage
/home/tony/work/inria/repo/swh/swh-environment/swh-indexer/swh/indexer/rehash.py:                ] = self.storage.content_get_metadata(  # noqa
/home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/swh/loader/package/npm/tests/test_npm.py:    contents_d = loader.storage.content_get_metadata(_expected_new_contents_first_visit)
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/browse/views/test_content.py:    content_data = archive_data.content_get_metadata(content["sha1"])
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/api/views/test_content.py:    expected_data = archive_data.content_get_metadata(content["sha1"])
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/data.py:    result = storage.content_get_metadata(contents)
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/common/test_service.py:    content_metadata = archive_data.content_get_metadata(content["sha1"])
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/common/test_service.py:    expected_content = archive_data.content_get_metadata(content["sha1"])
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/common/test_service.py:    expected_content = archive_data.content_get_metadata(content["sha1"])
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/conftest.py:    def content_get_metadata(self, cnt_id):
/home/tony/work/inria/repo/swh/swh-environment/swh-web/swh/web/tests/conftest.py:        metadata = self.storage.content_get_metadata([cnt_id_bytes])

Related to T645

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Aug 7 2020, 12:45 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 7 2020, 12:45 AM
swh-public-ci added a comment.Aug 7 2020, 12:51 AM

Build is green

Patch application report for D3733 (id=13139)

Could not rebase; Attempt merge onto 653b1f9ada...

Updating 653b1f9a..5994a08b
Fast-forward
 swh/storage/cassandra/storage.py    |  25 +++------
 swh/storage/in_memory.py            |  27 +++------
 swh/storage/interface.py            |  37 ++++--------
 swh/storage/objstorage.py           |  26 +++++----
 swh/storage/storage.py              |  29 ++++------
 swh/storage/tests/test_cassandra.py |   8 +--
 swh/storage/tests/test_filter.py    |   6 +-
 swh/storage/tests/test_retry.py     |  30 +++++-----
 swh/storage/tests/test_storage.py   | 109 ++++++++++++++----------------------
 9 files changed, 114 insertions(+), 183 deletions(-)
Changes applied before test
commit 5994a08b485d2cb0793fa9cac466c13266dc5769
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Aug 7 00:38:19 2020 +0200

    storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
    
    Related to T645

commit 8dc21c9b41903a598c08bcbcb114f7ac2e9b1489
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Aug 6 22:56:09 2020 +0200

    storage*: Rename content_get_data(Sha1) -> Optional[bytes]
    
    Rename the confusing endpoint `content_get` to `content_get_data`.
    This now works on one content as it is how it is used today.
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/709/ for more details.

Harbormaster completed remote builds in B14405: Diff 13139.Aug 7 2020, 12:51 AM
ardumont edited the summary of this revision. (Show Details)Aug 7 2020, 12:54 AM
ardumont edited the summary of this revision. (Show Details)Aug 7 2020, 12:59 AM
ardumont added inline comments.Aug 7 2020, 1:23 AM
swh/storage/cassandra/storage.py
219

Why do we pop the ctime again?

ardumont updated this revision to Diff 13143.Aug 7 2020, 1:38 AM

Realign cassandra/in-memory storages to pop the ctime as pg storage does

swh-public-ci added a comment.Aug 7 2020, 1:43 AM

Build is green

Patch application report for D3733 (id=13143)

Could not rebase; Attempt merge onto 653b1f9ada...

Updating 653b1f9a..b73aa66a
Fast-forward
 swh/storage/cassandra/storage.py    |  26 ++++-----
 swh/storage/in_memory.py            |  27 +++------
 swh/storage/interface.py            |  37 ++++--------
 swh/storage/objstorage.py           |  26 +++++----
 swh/storage/storage.py              |  29 ++++------
 swh/storage/tests/test_cassandra.py |   8 +--
 swh/storage/tests/test_filter.py    |   6 +-
 swh/storage/tests/test_retry.py     |  30 +++++-----
 swh/storage/tests/test_storage.py   | 109 ++++++++++++++----------------------
 9 files changed, 115 insertions(+), 183 deletions(-)
Changes applied before test
commit b73aa66a981183c9432c5c83dc3601626b358751
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Aug 7 00:38:19 2020 +0200

    storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
    
    Related to T645

commit 8dc21c9b41903a598c08bcbcb114f7ac2e9b1489
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Aug 6 22:56:09 2020 +0200

    storage*: Rename content_get_data(Sha1) -> Optional[bytes]
    
    Rename the confusing endpoint `content_get` to `content_get_data`.
    This now works on one content as it is how it is used today.
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/710/ for more details.

Harbormaster completed remote builds in B14409: Diff 13143.Aug 7 2020, 1:43 AM
ardumont edited the summary of this revision. (Show Details)Aug 7 2020, 6:41 AM
vlorentz accepted this revision.Aug 7 2020, 9:35 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/cassandra/storage.py
219

no idea, I just copied the pg storage's behavior.

This revision is now accepted and ready to land.Aug 7 2020, 9:35 AM
ardumont updated this revision to Diff 13157.Aug 7 2020, 11:09 AM

Rebase

swh-public-ci added a comment.Aug 7 2020, 11:15 AM

Build has FAILED

Patch application report for D3733 (id=13157)

Could not rebase; Attempt merge onto 653b1f9ada...

Updating 653b1f9a..fe6ffb86
Fast-forward
 requirements-swh.txt                |   2 +-
 swh/storage/cassandra/storage.py    |  27 ++++-----
 swh/storage/in_memory.py            |  27 ++++-----
 swh/storage/interface.py            |  36 +++---------
 swh/storage/objstorage.py           |  26 +++++----
 swh/storage/storage.py              |  29 +++++-----
 swh/storage/tests/test_cassandra.py |   8 +--
 swh/storage/tests/test_filter.py    |   6 +-
 swh/storage/tests/test_retry.py     |  30 +++++-----
 swh/storage/tests/test_storage.py   | 109 ++++++++++++++----------------------
 10 files changed, 118 insertions(+), 182 deletions(-)
Changes applied before test
commit fe6ffb865e6402239bd62ec91f6b9644c826ce11
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Aug 7 00:38:19 2020 +0200

    storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
    
    Related to T645

commit f057deddfdf16101ffbfaac1ee7725baf288d911
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Aug 6 22:56:09 2020 +0200

    storage*: Rename content_get_data(Sha1) -> Optional[bytes]
    
    Rename the confusing endpoint `content_get` to `content_get_data`.
    This now works on one content as it is how it is used today.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/712/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/712/console

Harbormaster failed remote builds in B14422: Diff 13157!Aug 7 2020, 11:15 AM
ardumont updated this revision to Diff 13159.Aug 7 2020, 12:28 PM

Rebase

swh-public-ci added a comment.Aug 7 2020, 12:36 PM

Build is green

Patch application report for D3733 (id=13159)

Could not rebase; Attempt merge onto 243e873175...

Updating 243e8731..d9ff3912
Fast-forward
 swh/storage/cassandra/storage.py    |  27 ++++-----
 swh/storage/in_memory.py            |  26 ++++-----
 swh/storage/interface.py            |  36 +++---------
 swh/storage/objstorage.py           |  26 +++++----
 swh/storage/storage.py              |  29 +++++-----
 swh/storage/tests/test_cassandra.py |   7 +--
 swh/storage/tests/test_filter.py    |   6 +-
 swh/storage/tests/test_retry.py     |  30 +++++-----
 swh/storage/tests/test_storage.py   | 106 ++++++++++++++----------------------
 9 files changed, 117 insertions(+), 176 deletions(-)
Changes applied before test
commit d9ff3912d5ab37dec24903b2f81c1dbcfa4b5eb7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Aug 7 00:38:19 2020 +0200

    storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]]
    
    Related to T645

commit bfa8f46ea44d05d5dce15071d164b368da3f975f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Aug 6 22:56:09 2020 +0200

    storage*: Rename content_get_data(Sha1) -> Optional[bytes]
    
    Rename the confusing endpoint `content_get` to `content_get_data`.
    This now works on one content as it is how it is used today.
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/715/ for more details.

Harbormaster completed remote builds in B14426: Diff 13159.Aug 7 2020, 12:36 PM
Closed by commit rDSTOd9ff3912d5ab: storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]] (authored by ardumont). · Explain WhyAug 7 2020, 12:37 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOd9ff3912d5ab: storage*: Rename and type content_get(List[Sha1]) -> List[Optional[Content]].

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
13 lines
13 lines
8 lines
16 lines
tests/
7 lines
14 lines
64 lines

Diff 13162

View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/interface.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
View Options

swh/storage/tests/test_retry.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal