Page MenuHomeSoftware Heritage

api client: add a stream_lines option
AbandonedPublic

Authored by seirl on Sep 27 2019, 4:35 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

Rationale: I need a way to query services that stream results line by line, like swh-graph. For that I need to access the iter_lines() function of requests' responses.

Diff Detail

Repository
rDCORE Foundations and core functionalities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 7983
Build 11503: tox-on-jenkinsJenkins
Build 11502: arc lint + arc unit

Event Timeline

Can you please try to:

  • explain the diff/commit's need (i guess it's swh-graph related)
  • add some tests around this (i believe there are a few tests around that part)

I'm a bit worried that this is a new option to existing methods that deal with msgpack input/output, when the data that you're retrieving is quite certainly not msgpack.

This kinda breaks the contract of this RPC API; but I guess the stream option did that already?

swh/core/api/__init__.py
140

So you're stripping leading newlines? That's a bit weird.

Do we care for non-ASCII lines? Should we try to get the expected encoding from the response headers instead?