Gracefully handle a client closing the connection in the middle of a response being streamed
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	olasd
	Apr 16 2021, 6:34 PM

Description

When requesting a streamed list of nodes from swh.graph, a naive client could decide to close the connection early (e.g. if it has found a node it was looking for).

In the current implementation, this makes the swh.graph aiohttp worker hang until the write to the closed socket times out (with a traceback). This takes several minutes, which makes it fairly easy to exhaust all aiohttp workers, and is a waste of resources in general.

We should figure out how to propagate the socket closure through all layers more eagerly.

Revisions and Commits

rDGRPH Compressed graph representation
	D7890	rDGRPH066edad7cc36 SwhGraphProperties: remove useless IOExceptions
	D7890	rDGRPHfd61a31ef0c4 Add CheckSwhid() and Stats() RPC methods
	D7890	rDGRPH0b4fd29e3b14 GRPC: initial commit with protobuf + java server
	D7890	rDGRPHb61ca2d1ef44 SwhUnidirectionalGraph: make constructor with properties public
	D7890	rDGRPH208f5dc60128 Java: add proto/ dir symlink
	D7890	rDGRPH431d98de6ae4 RPC: temporarily use loadLabelled() because loadLabelledMapped() doesn't work…
	D7890	rDGRPH822f03de2161 Java tests: migrate to GRPC
	D7890	rDGRPHf098232f4282 Python: migrate HTTP API to GRPC
	D7890	rDGRPH5c08b117f336 Remove old Java HTTP server, replaced by GRPC
	D7890	rDGRPHa4d1eb10793f Remove now useless backend.py
	D7890	rDGRPH9eaccde2cadb NodeIdMap: use new *MappedBigList classes, remove deprecated classes and MapFile
	D7890	rDGRPH6d63a39642ba Remove now useless dot.py
	D7890	rDGRPH4e592de704c7 Reorganize Python files (RPC/HTTP server distinction)
	D7890	rDGRPHccc6b295a387 requirements-test: add grpc-stubs
	D7890	rDGRPH131e7165ff72 mypy.ini: ignore grpc generated files
	D7890	rDGRPH5d36ca6eaffa flake: exclude swh/graph/rpc dir
	D7890	rDGRPHe7fb41e4cd99 Add rpc.StatsTest
	D7890	rDGRPH383ed27418f9 Add FindPathTo and FindPathBetween endpoints
	D7890	rDGRPHeefdd910c43c Remove GraphDirection.BOTH (labelled iteration not supported yet)
	D7890	rDGRPH3649bd634e61 proto: migrate to FieldMask to filter out fields
	D7890	rDGRPH70fc951eac99 Traversal: test impossible paths
	D7890	rDGRPH83be48e06673 Traversal: test traversals from multiple sources/to multiple dests
	D7890	rDGRPHa2c35364b57a Traversal: check for invalid arguments
	D7890	rDGRPHc117623b3c26 Traversal: simplify the StopTraversal logic
	D7890	rDGRPH60e93f13390c Remove CheckSwhid, use GetNode instead
	D7890	rDGRPH6e16ec9d7579 Add tests for CountNodes/CountEdges
	D7890	rDGRPH9f83ed730669 Traversal: add max depth and common ancestors tests
	D7890	rDGRPHc7559ac55f7c Document protobuf/grpc services and fields
	D7890	rDGRPHc241e13548f2 Traversal: add max edge tests
	D7890	rDGRPH328f26e31ccf More protobuf/grpc documentation, better field names
	D7890	rDGRPH2f149363d21f swhgraph.proto: small documentation fixes
	D7890	rDGRPHb58cb841b6a1 doc: First full draft of GRPC API documentation
	D7890	rDGRPH98fce8be99f5 Add inline docstrings to Java GRPC server
	D7890	rDGRPHe6e6866279da doc: add GRPC page skeleton
	D7890	rDGRPH3ce74dfd2a01 Regenerate python protobuf documentation
	D7890	rDGRPHa6e02f726926 style: typos and indent fixes
	D7890	rDGRPHab895e3a0b53 Remove old useless classes, including Traversal
	D7890	rDGRPHf528aa80db5b GetNode: make endpoint thread-safe with a lightweight copy of the graph
	D7890	rDGRPHd89f35478012 SwhBidirectionalGraph: fix copy() not actually copying subgraphs
	D7890	rDGRPH43a3e3302abe Update webgraph-big to 3.7.0
	D7890	rDGRPH53b7d553ba06 GraphServer: Use nproc threads in threadpool by default