Page MenuHomeSoftware Heritage

Gracefully handle a client closing the connection in the middle of a response being streamed
Closed, MigratedEdits Locked

Description

When requesting a streamed list of nodes from swh.graph, a naive client could decide to close the connection early (e.g. if it has found a node it was looking for).

In the current implementation, this makes the swh.graph aiohttp worker hang until the write to the closed socket times out (with a traceback). This takes several minutes, which makes it fairly easy to exhaust all aiohttp workers, and is a waste of resources in general.

We should figure out how to propagate the socket closure through all layers more eagerly.

Revisions and Commits

rDGRPH Compressed graph representation
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890
D7890

Event Timeline

olasd triaged this task as Normal priority.Apr 16 2021, 6:34 PM
olasd created this task.

GRPC update: cancelling a GRPC stream works fine, but it doesn't seem like aiohttp is doing that when the HTTP stream is closed.