Page MenuHomeSoftware Heritage

server: Group multiple object ids in the same HTTP chunk
ClosedPublic

Authored by vlorentz on Mar 1 2022, 3:18 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
Restricted Maniphest Task
Commits
rDOBJS9be723dcbc98: server: Group multiple object ids in the same HTTP chunk
Summary

This reduces the size of the response from 26 bytes per id
to an average of 20.06 bytes.

Depends on D7268.

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D7269 (id=26328)

Could not rebase; Attempt merge onto 76a5f36b41...

Updating 76a5f36..43e7d45
Fast-forward
 swh/objstorage/api/client.py              | 15 +++++------
 swh/objstorage/api/server.py              | 43 ++++---------------------------
 swh/objstorage/backends/pathslicing.py    | 16 ++++++++++++
 swh/objstorage/backends/seaweedfs/http.py |  4 +--
 4 files changed, 29 insertions(+), 49 deletions(-)
Changes applied before test
commit 43e7d45961ef5241f4157d864cf4218dba86b55e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:17:42 2022 +0100

    server: Group multiple object ids in the same HTTP chunk
    
    This reduces the size of the response from 26 bytes per id
    to an average of 20.06 bytes (rounded up to the closest
    multiple of 26).

commit ff319b60b2221333074d16659f8222a0d19bfe66
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:13:43 2022 +0100

    client: Do not depend on how the server chunks the response
    
    Currently, the server uses HTTP chunked encoding to send object ids
    one by one, each in its own HTTP frame.
    
    I think this is a mistake to rely on such a detail of the HTTP protocol
    in a high-level API like this.
    
    Additionally, a future commit will rewrite the server to use Flask
    instead of aiohttp, which does not allow this kind of fine-grained
    control about HTTP chunks.

commit b159f04b8d591947003cf3a4cd15fbcf0f3cb31e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 13:36:02 2022 +0100

    Remove method add_stream from the RPC API.
    
    Rationale:
    
    1. Only the pathslicing backend implements it
    2. No other package uses it (besides a dead code path in the vault)
    3. A future commit will rewrite the RPC server to use Flask instead of
       aiohttp, and rewriting this view correctly is going to be hard
       (though possible)

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/121/ for more details.

Build is green

Patch application report for D7269 (id=26348)

Could not rebase; Attempt merge onto 76a5f36b41...

Updating 76a5f36..9be723d
Fast-forward
 swh/objstorage/api/client.py              | 16 ++++++------
 swh/objstorage/api/server.py              | 43 ++++---------------------------
 swh/objstorage/backends/pathslicing.py    | 16 ++++++++++++
 swh/objstorage/backends/seaweedfs/http.py |  4 +--
 4 files changed, 30 insertions(+), 49 deletions(-)
Changes applied before test
commit 9be723dcbc9813a4a66e0068de22e36dbaee09c2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:17:42 2022 +0100

    server: Group multiple object ids in the same HTTP chunk
    
    This reduces the size of the response from 26 bytes per id
    to an average of 20.06 bytes (rounded up to the closest
    multiple of 26).

commit a2424b7a13e9a696329a9e9efa6c86f9864dd60b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 15:13:43 2022 +0100

    client: Do not depend on how the server chunks the response
    
    Currently, the server uses HTTP chunked encoding to send object ids
    one by one, each in its own HTTP frame.
    
    I think this is a mistake to rely on such a detail of the HTTP protocol
    in a high-level API like this.
    
    Additionally, a future commit will rewrite the server to use Flask
    instead of aiohttp, which does not allow this kind of fine-grained
    control about HTTP chunks.

commit b159f04b8d591947003cf3a4cd15fbcf0f3cb31e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Mar 1 13:36:02 2022 +0100

    Remove method add_stream from the RPC API.
    
    Rationale:
    
    1. Only the pathslicing backend implements it
    2. No other package uses it (besides a dead code path in the vault)
    3. A future commit will rewrite the RPC server to use Flask instead of
       aiohttp, and rewriting this view correctly is going to be hard
       (though possible)

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/131/ for more details.

olasd added a subscriber: olasd.

I really wish this had some kind of encapsulation/separator, but I guess with using raw bytes that's not really practical...

This revision is now accepted and ready to land.Mar 2 2022, 3:26 PM
vlorentz added a task: Restricted Maniphest Task.Mar 7 2022, 9:25 AM