Page MenuHomeSoftware Heritage

api/graph: Stream responses as in the proxied graph service
ClosedPublic

Authored by anlambert on Jan 7 2021, 6:58 PM.

Details

Summary

Ensure the responses that are streamed by the graph service are also streamed in
the Web API endpoint proxying it.

Closes T2900

Diff Detail

Repository
rDWAPPS Web applications
Branch
api-graph-streaming-response
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18229
Build 28137: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28136: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4824 (id=17078)

Rebasing onto 2ed38516a4...

Current branch diff-target is up to date.
Changes applied before test
commit c834a81ee608a9a1fb258de2d4a991526c197033
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Jan 7 18:57:15 2021 +0100

    api/graph: Stream responses as in the proxied graph service
    
    Ensure the responses that are streamed by the graph service are also streamed in
    the Web API endpoint proxying it.
    
    Closes T2900

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/547/ for more details.

Did you check this does not increase the total time of the request too much?

Did you check this does not increase the total time of the request too much?

Requests execution time seems similar using the streaming approach (see below, first curl command receives a streamed response, I used the costly request from T2900).

(swh) ✔ ~/swh/swh-environment/swh-web [api-graph-streaming-response L|✚ 1⚑ 28] 
12:56 $ time curl http://localhost:5004/api/1/graph/visit/edges/swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35/?edges=rev:rev >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 76.0M    0 76.0M    0     0  4226k      0 --:--:--  0:00:18 --:--:-- 4223k

real    0m18,438s
user    0m0,965s
sys     0m4,006s
(swh) ✔ ~/swh/swh-environment/swh-web [api-graph-streaming-response L|✚ 1⚑ 28] 
12:57 $ time curl -H "Authorization: Bearer ${TOKEN}" https://archive.softwareheritage.org/api/1/graph/visit/edges/swh:1:rev:f39d7d78b70e0f39facb1e4fab77ad3df5c52a35/?edges=rev:rev >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 76.0M  100 76.0M    0     0  3473k      0  0:00:22  0:00:22 --:--:-- 18.1M

real    0m22,432s
user    0m0,210s
sys     0m0,131s

Based on my understanding, that change is needed to improve swh-fuse.

This revision is now accepted and ready to land.Jan 8 2021, 2:00 PM

Build is green

Patch application report for D4824 (id=17089)

Rebasing onto 6f8b844070...

Current branch diff-target is up to date.
Changes applied before test
commit e605c3fa701ac3df3bddba0c9324444084db462d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Jan 7 18:57:15 2021 +0100

    api/graph: Stream responses as in the proxied graph service
    
    Ensure the responses that are streamed by the graph service are also streamed in
    the Web API endpoint proxying it.
    
    Closes T2900

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/548/ for more details.