Page MenuHomeSoftware Heritage
Differential D7277

Rename and document constants in objstorage.py
ClosedPublic
Actions

Authored by vlorentz on Mar 2 2022, 1:30 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDOBJS16bb0433c722: Rename and document constants in objstorage.py
Summary

Addresses D7268#inline-52102

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Mar 2 2022, 1:30 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 2 2022, 1:30 PM
vlorentz mentioned this in D7268: client: Do not depend on how the server chunks the response.Mar 2 2022, 1:30 PM
vlorentz updated this revision to Diff 26355.Mar 2 2022, 1:31 PM

remove useless (and typoed) alias

vlorentz updated this revision to Diff 26356.Mar 2 2022, 1:31 PM

ugh, wrong commits again

swh-public-ci added a comment.Mar 2 2022, 1:33 PM

Build is green

Patch application report for D7277 (id=26354)

Could not rebase; Attempt merge onto 76a5f36b41...

Updating 76a5f36..acfd8aa
Fast-forward
 requirements.txt                                   |   2 +-
 swh/objstorage/api/client.py                       |  76 ++-----
 swh/objstorage/api/server.py                       | 214 ++++++------------
 swh/objstorage/backends/in_memory.py               |  10 +-
 swh/objstorage/backends/pathslicing.py             |  22 +-
 swh/objstorage/backends/seaweedfs/http.py          |   4 +-
 swh/objstorage/cli.py                              |  20 +-
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        | 237 ++++++++++++++++++++
 .../multiplexer/multiplexer_objstorage.py          |   3 +-
 swh/objstorage/multiplexer/striping_objstorage.py  |   7 +-
 swh/objstorage/objstorage.py                       | 238 ++-------------------
 swh/objstorage/tests/objstorage_testing.py         |  32 +++
 swh/objstorage/tests/test_objstorage_api.py        |  10 +-
 .../tests/test_objstorage_pathslicing.py           |   4 +-
 15 files changed, 432 insertions(+), 449 deletions(-)
 create mode 100644 swh/objstorage/interface.py
Changes applied before test
commit acfd8aa851e0fe6b31e4b0f0ec946138e6d3a4d7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 2 13:24:27 2022 +0100

    Rename and document constants in objstorage.py

commit 33a786ce15c093c7ad9a9b040937850f0a51affd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 17:57:58 2022 +0100

    Add test to enforce all objstorage backends follow the interface

commit 32cfda0406aa2a8b9f4727abf2ea56b7563460ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:49:53 2022 +0100

    Use flask instead of aiohttp as RPC server.
    
    The azure backend cannot be run in the same thread as an asyncio
    event loop: https://forge.softwareheritage.org/T3981
    
    Additionally, this simplifies the code, by reusing the same
    method auto-generation as other SWH components.

commit 9be723dcbc9813a4a66e0068de22e36dbaee09c2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:17:42 2022 +0100

    server: Group multiple object ids in the same HTTP chunk
    
    This reduces the size of the response from 26 bytes per id
    to an average of 20.06 bytes (rounded up to the closest
    multiple of 26).

commit a2424b7a13e9a696329a9e9efa6c86f9864dd60b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:13:43 2022 +0100

    client: Do not depend on how the server chunks the response
    
    Currently, the server uses HTTP chunked encoding to send object ids
    one by one, each in its own HTTP frame.
    
    I think this is a mistake to rely on such a detail of the HTTP protocol
    in a high-level API like this.
    
    Additionally, a future commit will rewrite the server to use Flask
    instead of aiohttp, which does not allow this kind of fine-grained
    control about HTTP chunks.

commit b159f04b8d591947003cf3a4cd15fbcf0f3cb31e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 13:36:02 2022 +0100

    Remove method add_stream from the RPC API.
    
    Rationale:
    
    1. Only the pathslicing backend implements it
    2. No other package uses it (besides a dead code path in the vault)
    3. A future commit will rewrite the RPC server to use Flask instead of
       aiohttp, and rewriting this view correctly is going to be hard
       (though possible)

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/135/ for more details.

Harbormaster completed remote builds in B27229: Diff 26354.Mar 2 2022, 1:33 PM
swh-public-ci added a comment.Mar 2 2022, 1:34 PM

Build is green

Patch application report for D7277 (id=26355)

Rebasing onto 76a5f36b41...

Current branch diff-target is up to date.
Changes applied before test
commit 16bb0433c722c8cf5212b48e5ead1eece8cce5a8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 2 13:24:27 2022 +0100

    Rename and document constants in objstorage.py

commit 33a786ce15c093c7ad9a9b040937850f0a51affd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 17:57:58 2022 +0100

    Add test to enforce all objstorage backends follow the interface

commit 32cfda0406aa2a8b9f4727abf2ea56b7563460ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:49:53 2022 +0100

    Use flask instead of aiohttp as RPC server.
    
    The azure backend cannot be run in the same thread as an asyncio
    event loop: https://forge.softwareheritage.org/T3981
    
    Additionally, this simplifies the code, by reusing the same
    method auto-generation as other SWH components.

commit 9be723dcbc9813a4a66e0068de22e36dbaee09c2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:17:42 2022 +0100

    server: Group multiple object ids in the same HTTP chunk
    
    This reduces the size of the response from 26 bytes per id
    to an average of 20.06 bytes (rounded up to the closest
    multiple of 26).

commit a2424b7a13e9a696329a9e9efa6c86f9864dd60b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:13:43 2022 +0100

    client: Do not depend on how the server chunks the response
    
    Currently, the server uses HTTP chunked encoding to send object ids
    one by one, each in its own HTTP frame.
    
    I think this is a mistake to rely on such a detail of the HTTP protocol
    in a high-level API like this.
    
    Additionally, a future commit will rewrite the server to use Flask
    instead of aiohttp, which does not allow this kind of fine-grained
    control about HTTP chunks.

commit b159f04b8d591947003cf3a4cd15fbcf0f3cb31e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 13:36:02 2022 +0100

    Remove method add_stream from the RPC API.
    
    Rationale:
    
    1. Only the pathslicing backend implements it
    2. No other package uses it (besides a dead code path in the vault)
    3. A future commit will rewrite the RPC server to use Flask instead of
       aiohttp, and rewriting this view correctly is going to be hard
       (though possible)

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/136/ for more details.

Harbormaster completed remote builds in B27230: Diff 26355.Mar 2 2022, 1:34 PM
swh-public-ci added a comment.Mar 2 2022, 1:34 PM

Build is green

Patch application report for D7277 (id=26356)

Could not rebase; Attempt merge onto 76a5f36b41...

Updating 76a5f36..16bb043
Fast-forward
 requirements.txt                                   |   2 +-
 swh/objstorage/api/client.py                       |  76 ++-----
 swh/objstorage/api/server.py                       | 214 ++++++-------------
 swh/objstorage/backends/in_memory.py               |  10 +-
 swh/objstorage/backends/pathslicing.py             |  22 +-
 swh/objstorage/backends/seaweedfs/http.py          |   4 +-
 swh/objstorage/cli.py                              |  20 +-
 swh/objstorage/factory.py                          |   2 +-
 swh/objstorage/interface.py                        | 237 +++++++++++++++++++++
 .../multiplexer/multiplexer_objstorage.py          |   3 +-
 swh/objstorage/multiplexer/striping_objstorage.py  |   7 +-
 swh/objstorage/objstorage.py                       | 235 ++------------------
 swh/objstorage/tests/objstorage_testing.py         |  32 +++
 swh/objstorage/tests/test_objstorage_api.py        |  10 +-
 .../tests/test_objstorage_pathslicing.py           |   4 +-
 15 files changed, 429 insertions(+), 449 deletions(-)
 create mode 100644 swh/objstorage/interface.py
Changes applied before test
commit 16bb0433c722c8cf5212b48e5ead1eece8cce5a8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 2 13:24:27 2022 +0100

    Rename and document constants in objstorage.py

commit 33a786ce15c093c7ad9a9b040937850f0a51affd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 17:57:58 2022 +0100

    Add test to enforce all objstorage backends follow the interface

commit 32cfda0406aa2a8b9f4727abf2ea56b7563460ea
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:49:53 2022 +0100

    Use flask instead of aiohttp as RPC server.
    
    The azure backend cannot be run in the same thread as an asyncio
    event loop: https://forge.softwareheritage.org/T3981
    
    Additionally, this simplifies the code, by reusing the same
    method auto-generation as other SWH components.

commit 9be723dcbc9813a4a66e0068de22e36dbaee09c2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:17:42 2022 +0100

    server: Group multiple object ids in the same HTTP chunk
    
    This reduces the size of the response from 26 bytes per id
    to an average of 20.06 bytes (rounded up to the closest
    multiple of 26).

commit a2424b7a13e9a696329a9e9efa6c86f9864dd60b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:13:43 2022 +0100

    client: Do not depend on how the server chunks the response
    
    Currently, the server uses HTTP chunked encoding to send object ids
    one by one, each in its own HTTP frame.
    
    I think this is a mistake to rely on such a detail of the HTTP protocol
    in a high-level API like this.
    
    Additionally, a future commit will rewrite the server to use Flask
    instead of aiohttp, which does not allow this kind of fine-grained
    control about HTTP chunks.

commit b159f04b8d591947003cf3a4cd15fbcf0f3cb31e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 13:36:02 2022 +0100

    Remove method add_stream from the RPC API.
    
    Rationale:
    
    1. Only the pathslicing backend implements it
    2. No other package uses it (besides a dead code path in the vault)
    3. A future commit will rewrite the RPC server to use Flask instead of
       aiohttp, and rewriting this view correctly is going to be hard
       (though possible)

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/137/ for more details.

Harbormaster completed remote builds in B27231: Diff 26356.Mar 2 2022, 1:34 PM
vlorentz requested review of this revision.Mar 2 2022, 1:34 PM
olasd accepted this revision.Mar 2 2022, 4:57 PM
olasd added a subscriber: olasd.

Great, thanks!

This revision is now accepted and ready to land.Mar 2 2022, 4:57 PM
Closed by commit rDOBJS16bb0433c722: Rename and document constants in objstorage.py (authored by vlorentz). · Explain WhyMar 2 2022, 5:25 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDOBJS16bb0433c722: Rename and document constants in objstorage.py.

Revision Contents
Changeset List

PathSize
swh/
objstorage/
api/
10 lines
backends/
6 lines
2 lines
13 lines
tests/
4 lines

Diff 26363

View Options

swh/objstorage/api/client.py

Loading...
View Options

swh/objstorage/backends/pathslicing.py

Loading...
View Options

swh/objstorage/factory.py

Loading...
View Options

swh/objstorage/objstorage.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_pathslicing.py

Loading...
About · Developer information · Legal