Page MenuHomeSoftware Heritage
Differential D8014

Add type annotations to all objstorage backends
ClosedPublic
Actions

Authored by vlorentz on Jun 21 2022, 6:16 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDOBJS68b2be94d8af: Add type annotations to all objstorage backends
rDOBJS56216ab438cb: libcloud: Default to gzip compression
Summary

Also makes libcloud default to gzip compression like other backends,
which keeps mypy happy once annotations are added

w/ @olasd

Depends on D7999

Diff Detail

Repository
rDOBJS Object storage
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29963
Build 46845: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46844: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jun 21 2022, 6:16 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 21 2022, 6:16 PM
swh-public-ci added a comment.Jun 21 2022, 6:16 PM

Build has FAILED

Patch application report for D8014 (id=28874)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..212381b
Fast-forward
 swh/objstorage/api/client.py                       |  12 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++-
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  39 +++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 +++
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 +--
 swh/objstorage/objstorage.py                       | 115 ++++++++++-----------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 23 files changed, 251 insertions(+), 257 deletions(-)
 create mode 100644 swh/objstorage/constants.py
Changes applied before test
commit 212381b93a75f09c64975e294d235d32ea4e5414
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/153/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/153/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 21 2022, 6:16 PM
Harbormaster failed remote builds in B29961: Diff 28874!
vlorentz edited the summary of this revision. (Show Details)Jun 21 2022, 6:18 PM
vlorentz edited the summary of this revision. (Show Details)
vlorentz added a parent revision: D7999: crates: Fix incorrect documentation of release format.
vlorentz added a subscriber: olasd.
vlorentz updated this revision to Diff 28875.Jun 21 2022, 6:21 PM

remove unused imports

swh-public-ci added a comment.Jun 21 2022, 6:23 PM

Build has FAILED

Patch application report for D8014 (id=28875)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..1140fd3
Fast-forward
 swh/objstorage/api/client.py                       |  14 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++--
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  49 +++++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 ++++
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 ++--
 swh/objstorage/objstorage.py                       | 110 +++++++++------------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 .../tests/test_objstorage_pathslicing.py           |   2 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 25 files changed, 251 insertions(+), 268 deletions(-)
 create mode 100644 swh/objstorage/constants.py
Changes applied before test
commit 1140fd3bbdd48de8f7ebb6fc66794384e835a73d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/154/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/154/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 21 2022, 6:23 PM
Harbormaster failed remote builds in B29962: Diff 28875!
vlorentz updated this revision to Diff 28876.Jun 21 2022, 6:23 PM

fix py3.7 support

swh-public-ci added a comment.Jun 21 2022, 6:26 PM

Build is green

Patch application report for D8014 (id=28876)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..68b2be9
Fast-forward
 swh/objstorage/api/client.py                       |  14 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++--
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  49 +++++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 ++++
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 ++--
 swh/objstorage/objstorage.py                       | 110 +++++++++------------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 .../tests/test_objstorage_pathslicing.py           |   2 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 25 files changed, 251 insertions(+), 268 deletions(-)
 create mode 100644 swh/objstorage/constants.py
Changes applied before test
commit 68b2be94d8af30464350c0858fce5a55ef25daf8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/155/ for more details.

Harbormaster completed remote builds in B29963: Diff 28876.Jun 21 2022, 6:26 PM
vlorentz requested review of this revision.Jun 21 2022, 6:26 PM
vlorentz added a child revision: D8017: Make add() and restore() return None instead of ObjId.Jun 22 2022, 11:19 AM
olasd accepted this revision.Jun 22 2022, 4:01 PM

The factory type annotations are in D8024.

This revision is now accepted and ready to land.Jun 22 2022, 4:01 PM
Closed by commit rDOBJS56216ab438cb: libcloud: Default to gzip compression (authored by vlorentz). · Explain WhyJun 22 2022, 4:04 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDOBJS56216ab438cb: libcloud: Default to gzip compression.
vlorentz added a commit: rDOBJS68b2be94d8af: Add type annotations to all objstorage backends.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
objstorage/
api/
14 lines100%
Loading...
backends/
15 lines97%
Loading...
8 lines92%
Loading...
18 lines85%
Loading...
12 lines96%
Loading...
13 lines91%
Loading...
46 lines91%
Loading...
seaweedfs/
27 lines87%
Loading...
winery/
15 lines72%
Loading...
17 lines100%
Loading...
2 lines76%
Loading...
60 lines69%
Loading...
multiplexer/
17 lines96%
Loading...
104 lines--
tests/
2 lines98%
Loading...

Diff 28876

View Options

swh/objstorage/api/client.py

Loading...
View Options

swh/objstorage/backends/azure.py

Loading...
View Options

swh/objstorage/backends/generator.py

Loading...
View Options

swh/objstorage/backends/http.py

Loading...
View Options

swh/objstorage/backends/in_memory.py

Loading...
View Options

swh/objstorage/backends/libcloud.py

Loading...
View Options

swh/objstorage/backends/pathslicing.py

Loading...
View Options

swh/objstorage/backends/seaweedfs/objstorage.py

Loading...
View Options

swh/objstorage/backends/winery/objstorage.py

Loading...
View Options

swh/objstorage/constants.py

Loading...
View Options

swh/objstorage/factory.py

Loading...
View Options

swh/objstorage/interface.py

Loading...
View Options

swh/objstorage/multiplexer/multiplexer_objstorage.py

Loading...
View Options

swh/objstorage/objstorage.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_pathslicing.py

Loading...
About · Developer information · Legal