Page MenuHomeSoftware Heritage

Add support for querying raw-extrinsic-metadata for origins
ClosedPublic

Authored by vlorentz on Jul 12 2022, 1:49 PM.

Details

Summary

It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
endpoint directly, as extended SWHIDs should not be part of the public API.

Instead, clients should query api-1-origin first, and rely on the link
in its response to fetch metadata.

Event Timeline

Build is green

Patch application report for D8114 (id=29307)

Could not rebase; Attempt merge onto 4e388ee956...

Updating 4e388ee9..11123d28
Fast-forward
 pyproject.toml                           |  1 +
 requirements-swh.txt                     |  2 +-
 swh/web/api/utils.py                     |  8 ++++++-
 swh/web/api/views/metadata.py            | 41 ++++++++++++++++++++++++++++----
 swh/web/api/views/origin.py              |  4 ++++
 swh/web/tests/api/test_utils.py          | 21 ++++++++++++++--
 swh/web/tests/api/views/test_metadata.py | 31 +++++++-----------------
 7 files changed, 77 insertions(+), 31 deletions(-)
Changes applied before test
commit 11123d283d3ca691657be51d2dd46c71de58a66c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 13:49:37 2022 +0200

    Add support for querying raw-extrinsic-metadata for origins
    
    It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
    endpoint directly, as extended SWHIDs should not be part of the public API.
    
    Instead, clients should query api-1-origin first, and rely on the link
    in its response to fetch metadata.

commit 78024a5cad824189c399fe7b6428b577134c53e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 12:31:15 2022 +0200

    Exclude highlightjs test files from Black
    
    Black reforms `word2vec.ipynb` otherwise (or complains the ipynb is not
    installed)

commit e61a0dbb76ca67f31b2c84eec62d5fd3b5a22b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 11:58:08 2022 +0200

    test_metadata: Reuse swh-model's raw_extrinsic_metadata() strategy
    
    It now allows setting custom strategies as 'target'

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1921/ for more details.

better filename escape + add test

This revision was not accepted when it landed; it landed in state Needs Review.Jul 12 2022, 2:32 PM
This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.

uh what, pushed the wrong commit

Build is green

Patch application report for D8114 (id=29309)

Rebasing onto 4e388ee956...

Current branch diff-target is up to date.
Changes applied before test
commit 5b02f0e928a2cfb24b507e18118a56fe9e236857
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 13:49:37 2022 +0200

    Add support for querying raw-extrinsic-metadata for origins
    
    It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
    endpoint directly, as extended SWHIDs should not be part of the public API.
    
    Instead, clients should query api-1-origin first, and rely on the link
    in its response to fetch metadata.

commit 78024a5cad824189c399fe7b6428b577134c53e4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 12:31:15 2022 +0200

    Exclude highlightjs test files from Black
    
    Black reforms `word2vec.ipynb` otherwise (or complains the ipynb is not
    installed)

commit e61a0dbb76ca67f31b2c84eec62d5fd3b5a22b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 11:58:08 2022 +0200

    test_metadata: Reuse swh-model's raw_extrinsic_metadata() strategy
    
    It now allows setting custom strategies as 'target'

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1923/ for more details.

Build is green

Patch application report for D8114 (id=29314)

Rebasing onto 78024a5cad...

Current branch diff-target is up to date.
Changes applied before test
commit 5b02f0e928a2cfb24b507e18118a56fe9e236857
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 13:49:37 2022 +0200

    Add support for querying raw-extrinsic-metadata for origins
    
    It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
    endpoint directly, as extended SWHIDs should not be part of the public API.
    
    Instead, clients should query api-1-origin first, and rely on the link
    in its response to fetch metadata.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1924/ for more details.

anlambert added inline comments.
swh/web/api/views/metadata.py
140

Why a tuple here ?

153

HATEOAS ?

swh/web/tests/api/views/test_metadata.py
59

any reason to restrict the number of examples ?

swh/web/api/views/metadata.py
140

Why not? It is guaranteed to have a single result

153

by following the URL from the other API endpoint, instead of computing the URL directly

swh/web/tests/api/views/test_metadata.py
59

I don't see the point of testing more; the origin URL is always the same anyway

update comment to mention api-1-origin explicitly

Looks good to me.

swh/web/api/views/metadata.py
140

Oh I see, this is for extracting the first element of the list, I am not used to this syntax so my confusion here.

swh/web/tests/api/views/test_metadata.py
59

ack

This revision is now accepted and ready to land.Jul 12 2022, 5:14 PM
swh/web/api/views/metadata.py
140

only element of the list (to extract the first, it would be (first, *rest) = ...)

Build is green

Patch application report for D8114 (id=29327)

Rebasing onto b80d078ee1...

First, rewinding head to replay your work on top of it...
Applying: Add support for querying raw-extrinsic-metadata for origins
Changes applied before test
commit 80ff1403f6e0f89ae04ef9d368bb8abd1fb3b37a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 13:49:37 2022 +0200

    Add support for querying raw-extrinsic-metadata for origins
    
    It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
    endpoint directly, as extended SWHIDs should not be part of the public API.
    
    Instead, clients should query api-1-origin first, and rely on the link
    in its response to fetch metadata.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1928/ for more details.

This revision was landed with ongoing or failed builds.Jul 13 2022, 11:01 AM
This revision was automatically updated to reflect the committed changes.

Build was aborted

Patch application report for D8114 (id=29340)

Rebasing onto 6f9d36afe4...

Current branch diff-target is up to date.
Changes applied before test
commit b76a3cec0dc71bd46a6c3f89dc26f00734b6a706
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Jul 12 13:49:37 2022 +0200

    Add support for querying raw-extrinsic-metadata for origins
    
    It is purposefully discouraged to query the api-1-raw-extrinsic-metadata-swhid
    endpoint directly, as extended SWHIDs should not be part of the public API.
    
    Instead, clients should query api-1-origin first, and rely on the link
    in its response to fetch metadata.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1933/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1933/console