Page MenuHomeSoftware Heritage
Differential D4077

api: Add /graph endpoint proxying Software Heritage Graph service
ClosedPublic
Actions

Authored by anlambert on Sep 29 2020, 2:10 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2589: expose swh-graph API at archive.s.o/api/1/graph/
Commits
rDWAPPS2d69cbc46a16: api: Add /graph endpoint proxying Software Heritage Graph service
Summary

Add a new /graph endpoint to the Web API acting as a proxy for the Software
Heritage Graph service. It forwards graph queries to it and return its raw
responses.

The endpoint requires authentication and user special permission in order
to be able to query it.

It also adds an extra query parameter "resolve_origins" enabling to resolve
origin urls from their sha1 representations returned by the Graph service.

Closes T2589

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Sep 29 2020, 2:10 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 29 2020, 2:10 PM
anlambert updated this revision to Diff 14391.Sep 29 2020, 2:12 PM

Update: Remove empty line

swh-public-ci added a comment.Sep 29 2020, 2:17 PM

Build is green

Patch application report for D4077 (id=14389)

Rebasing onto 504f0db368...

Current branch diff-target is up to date.
Changes applied before test
commit a210df3f20748bfebfb3ec1ff6de2cab66b9765c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Sep 29 14:08:05 2020 +0200

    api: Add /graph endpoint proxying Software Heritage Graph service
    
    Add a new /graph endpoint to the Web API acting as a proxy for the Software
    Heritage Graph service. It forwards graph queries to it and return its raw
    responses.
    
    The endpoint requires authentication and user special permission in order
    to be able to query it.
    
    It also adds an extra query parameter "resolve_origins" enabling to resolve
    origin urls from their sha1 representations returned by the Graph service.
    
    Closes T2589

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/394/ for more details.

Harbormaster completed remote builds in B15684: Diff 14389.Sep 29 2020, 2:17 PM
swh-public-ci added a comment.Sep 29 2020, 2:24 PM

Build is green

Patch application report for D4077 (id=14391)

Rebasing onto 504f0db368...

Current branch diff-target is up to date.
Changes applied before test
commit 8ae5e23a7a38eab23f652379bc572e1d01f57fac
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Sep 29 14:08:05 2020 +0200

    api: Add /graph endpoint proxying Software Heritage Graph service
    
    Add a new /graph endpoint to the Web API acting as a proxy for the Software
    Heritage Graph service. It forwards graph queries to it and return its raw
    responses.
    
    The endpoint requires authentication and user special permission in order
    to be able to query it.
    
    It also adds an extra query parameter "resolve_origins" enabling to resolve
    origin urls from their sha1 representations returned by the Graph service.
    
    Closes T2589

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/395/ for more details.

Harbormaster completed remote builds in B15687: Diff 14391.Sep 29 2020, 2:24 PM
anlambert updated this revision to Diff 14395.Sep 29 2020, 2:28 PM

Update docs

swh-public-ci added a comment.Sep 29 2020, 2:36 PM

Build is green

Patch application report for D4077 (id=14395)

Rebasing onto 504f0db368...

Current branch diff-target is up to date.
Changes applied before test
commit 6b8fceaecd0c05d36e661de8d0d143d06f1a3f11
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Sep 29 14:08:05 2020 +0200

    api: Add /graph endpoint proxying Software Heritage Graph service
    
    Add a new /graph endpoint to the Web API acting as a proxy for the Software
    Heritage Graph service. It forwards graph queries to it and return its raw
    responses.
    
    The endpoint requires authentication and user special permission in order
    to be able to query it.
    
    It also adds an extra query parameter "resolve_origins" enabling to resolve
    origin urls from their sha1 representations returned by the Graph service.
    
    Closes T2589

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/396/ for more details.

Harbormaster completed remote builds in B15689: Diff 14395.Sep 29 2020, 2:36 PM
vlorentz requested changes to this revision.Sep 29 2020, 3:39 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/web/api/apidoc.py
217–219

why this change?

swh/web/api/views/graph.py
26

it could use a docstring (and/or a better name)

30

this line isn't covered, could you add a test?

63

same

127

don't need an f-string here

swh/web/tests/api/views/test_graph.py
55–71

alternatively, a trick I learned recently:

response_text = textwrap.dedent(
    """\
    swh:1:cnt:1d3dace0a825b0535c37c53ed669ef817e9c1b47
    ...
    swh:1:cnt:f045ee845c7f14d903a2c035b2691a7c400c01f0
    """
)

it spares the extra quotes and \n on each line.

But either way is fine

125–132

textwrap.dedent could be nice here too

This revision now requires changes to proceed.Sep 29 2020, 3:39 PM
anlambert added inline comments.Sep 29 2020, 3:50 PM
swh/web/api/apidoc.py
217–219

To support multiple api url examples in endpoint description. Only one example was used in other endpoint docstrings until then.

swh/web/api/views/graph.py
30

Where can I see the detailed coverage report for Python code ? Jenkins only shows graphs to me on the coverage report page: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/396/coverage/

127

yes it is (notice the ?)

swh/web/tests/api/views/test_graph.py
55–71

nice !

vlorentz added inline comments.Sep 29 2020, 4:06 PM
swh/web/api/apidoc.py
217–219

but I don't see why the old regexp didn't work for multiple examples. (.*) shouldn't match multiple lines by default

swh/web/api/views/graph.py
30

Phabricator shows it with colors on the right side of the diff

127

"?" + request.GET.urlencode(safe="/;:") is simpler IMO, but ok

anlambert added inline comments.Sep 29 2020, 4:42 PM
swh/web/api/apidoc.py
217–219

Ok got it, this works:

examples_str = re.sub(".*`(.+)`.*", r"/api/1/\1", text)
self.data["examples"] += examples_str.split("\n")
swh/web/api/views/graph.py
30

Jeez ! Thanks, I never noticed it.

anlambert updated this revision to Diff 14403.Sep 29 2020, 5:06 PM

Update: Address @vlorentz comments

swh-public-ci added a comment.Sep 29 2020, 5:13 PM

Build is green

Patch application report for D4077 (id=14403)

Rebasing onto 504f0db368...

Current branch diff-target is up to date.
Changes applied before test
commit 2d69cbc46a1655d88aed7a935e946a3cfa528899
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Sep 29 14:08:05 2020 +0200

    api: Add /graph endpoint proxying Software Heritage Graph service
    
    Add a new /graph endpoint to the Web API acting as a proxy for the Software
    Heritage Graph service. It forwards graph queries to it and return its raw
    responses.
    
    The endpoint requires authentication and user special permission in order
    to be able to query it.
    
    It also adds an extra query parameter "resolve_origins" enabling to resolve
    origin urls from their sha1 representations returned by the Graph service.
    
    Closes T2589

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/397/ for more details.

Harbormaster completed remote builds in B15697: Diff 14403.Sep 29 2020, 5:13 PM
vlorentz accepted this revision.Sep 29 2020, 6:16 PM
This revision is now accepted and ready to land.Sep 29 2020, 6:16 PM
Closed by commit rDWAPPS2d69cbc46a16: api: Add /graph endpoint proxying Software Heritage Graph service (authored by anlambert). · Explain WhySep 30 2020, 11:05 AM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPS2d69cbc46a16: api: Add /graph endpoint proxying Software Heritage Graph service.

Revision Contents
Changeset List

PathSize
docs/
4 lines
2 lines
1 line
swh/
web/
api/
7 lines
15 lines
1 line
views/
144 lines
4 lines
tests/
api/
views/
245 lines

Diff 14439

View Options

docs/uri-scheme-api-graph.rst

Loading...
View Options

docs/uri-scheme-api.rst

Loading...
View Options

pytest.ini

Loading...
View Options

swh/web/api/apidoc.py

Loading...
View Options

swh/web/api/renderers.py

Loading...
View Options

swh/web/api/urls.py

Loading...
View Options

swh/web/api/views/graph.py

Loading...
View Options

swh/web/config.py

Loading...
View Options

swh/web/tests/api/views/test_graph.py

Loading...
About · Developer information · Legal