Add type annotations to all objstorage backends
ClosedPublic
Actions

Authored by vlorentz on Jun 21 2022, 6:16 PM.

Details

Reviewers

Group Reviewers

Commits

rDOBJS68b2be94d8af: Add type annotations to all objstorage backends
rDOBJS56216ab438cb: libcloud: Default to gzip compression

Summary

Also makes libcloud default to gzip compression like other backends,
which keeps mypy happy once annotations are added

w/ @olasd

Depends on D7999

Diff Detail

Repository

rDOBJS Object storage

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jun 21 2022, 6:16 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 21 2022, 6:16 PM

Build has FAILED

Patch application report for D8014 (id=28874)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..212381b
Fast-forward
 swh/objstorage/api/client.py                       |  12 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++-
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  39 +++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 +++
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 +--
 swh/objstorage/objstorage.py                       | 115 ++++++++++-----------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 23 files changed, 251 insertions(+), 257 deletions(-)
 create mode 100644 swh/objstorage/constants.py

Changes applied before test

commit 212381b93a75f09c64975e294d235d32ea4e5414
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/153/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/153/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 21 2022, 6:16 PM

Harbormaster failed remote builds in B29961: Diff 28874!

vlorentz edited the summary of this revision. (Show Details)Jun 21 2022, 6:18 PM

vlorentz edited the summary of this revision. (Show Details)

vlorentz added a parent revision: D7999: crates: Fix incorrect documentation of release format.

vlorentz added a subscriber: olasd.

remove unused imports

Build has FAILED

Patch application report for D8014 (id=28875)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..1140fd3
Fast-forward
 swh/objstorage/api/client.py                       |  14 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++--
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  49 +++++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 ++++
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 ++--
 swh/objstorage/objstorage.py                       | 110 +++++++++------------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 .../tests/test_objstorage_pathslicing.py           |   2 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 25 files changed, 251 insertions(+), 268 deletions(-)
 create mode 100644 swh/objstorage/constants.py

Changes applied before test

commit 1140fd3bbdd48de8f7ebb6fc66794384e835a73d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/154/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/154/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 21 2022, 6:23 PM

Harbormaster failed remote builds in B29962: Diff 28875!

fix py3.7 support

Build is green

Patch application report for D8014 (id=28876)

Could not rebase; Attempt merge onto 2c7e49136a...

Updating 2c7e491..68b2be9
Fast-forward
 swh/objstorage/api/client.py                       |  14 ++-
 swh/objstorage/backends/azure.py                   |  23 ++---
 swh/objstorage/backends/generator.py               |  10 +-
 swh/objstorage/backends/http.py                    |  18 ++--
 swh/objstorage/backends/in_memory.py               |  15 ++-
 swh/objstorage/backends/libcloud.py                |  17 ++--
 swh/objstorage/backends/noop.py                    |   2 +-
 swh/objstorage/backends/pathslicing.py             |  49 +++++----
 swh/objstorage/backends/seaweedfs/objstorage.py    |  31 +++---
 swh/objstorage/backends/winery/objstorage.py       |  31 ++----
 swh/objstorage/cli.py                              |   8 +-
 swh/objstorage/constants.py                        |  17 ++++
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        |  64 ++++++------
 swh/objstorage/multiplexer/filter/filter.py        |   4 +-
 swh/objstorage/multiplexer/filter/id_filter.py     |   9 +-
 .../multiplexer/multiplexer_objstorage.py          |  17 ++--
 swh/objstorage/objstorage.py                       | 110 +++++++++------------
 swh/objstorage/tests/objstorage_testing.py         |   6 --
 swh/objstorage/tests/test_multiplexer_filter.py    |  48 +++++----
 swh/objstorage/tests/test_objstorage_http.py       |  10 +-
 .../tests/test_objstorage_multiplexer.py           |   4 +-
 .../tests/test_objstorage_pathslicing.py           |   2 +-
 swh/objstorage/tests/test_objstorage_striping.py   |   3 -
 swh/objstorage/tests/test_objstorage_winery.py     |   5 +-
 25 files changed, 251 insertions(+), 268 deletions(-)
 create mode 100644 swh/objstorage/constants.py

Changes applied before test

commit 68b2be94d8af30464350c0858fce5a55ef25daf8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 17:56:55 2022 +0200

    Add type annotations to all objstorage backends

commit 56216ab438cb0620ccc3fd5b6173d8d4e5aff4fa
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 18:05:44 2022 +0200

    libcloud: Default to gzip compression
    
    Like other backends.
    
    Allowing None will be an issue once we add type annotations to
    methods which use self.compression.

commit 6cdfe555fab7f75423bae7d820b2cd25ef4f4530
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:05:45 2022 +0200

    Make obj_id argument of ObjStorage.restore() required
    
    For consistency with ObjStorage.add()

commit 99e0d40bb0d2f30dc071ad23692a661b348171fc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jun 21 16:00:43 2022 +0200

    Make obj_id argument of ObjStorage.add() required
    
    In practice, it is not very useful to recompute it in the objstorage,
    as callers already know it.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/155/ for more details.

Harbormaster completed remote builds in B29963: Diff 28876.Jun 21 2022, 6:26 PM

vlorentz requested review of this revision.Jun 21 2022, 6:26 PM

vlorentz added a child revision: D8017: Make add() and restore() return None instead of ObjId.Jun 22 2022, 11:19 AM

The factory type annotations are in D8024.

This revision is now accepted and ready to land.Jun 22 2022, 4:01 PM

Closed by commit rDOBJS56216ab438cb: libcloud: Default to gzip compression (authored by vlorentz). · Explain WhyJun 22 2022, 4:04 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDOBJS56216ab438cb: libcloud: Default to gzip compression.

vlorentz added a commit: rDOBJS68b2be94d8af: Add type annotations to all objstorage backends.