Page MenuHomeSoftware Heritage

api/raw: Restrict raw object endpoint use to privileged users
ClosedPublic

Authored by anlambert on May 9 2022, 4:30 PM.

Details

Summary

As that endpoint can leak sensible information regarding authors in
the archive, it has been decided to not make it publicly available.

So restrict its use to staff users and users with new permission
swh.web.api.raw_object.

Also lift rate limit on that new endpoint to ease development
of the bridge of Software Heritage to IPFS.

Related to T4226

Diff Detail

Repository
rDWAPPS Web applications
Branch
throttling-raw-object-endpoint-exception
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29228
Build 45697: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 45696: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7794 (id=28130)

Rebasing onto b30dc0510f...

Current branch diff-target is up to date.
Changes applied before test
commit 3c103645ea376c7cb163610cb77eb23f7558c636
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon May 9 16:22:59 2022 +0200

    api/throttling: Lift rate limit for raw endpoint with permission
    
    Add a new user permission "swh.web.api.raw_object" enabling to lift
    rate limit on the /api/1/raw/(swhid)/ endpoint.
    
    It will be used by the Software Heritage bridge to IPFS developped
    by Obsidian Systems.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1806/ for more details.

ardumont added a subscriber: ardumont.

lgtm

Can you please add the associated task id in the diff description?
(so one can browse to the related task more easily ;) ?

TIA

This revision is now accepted and ready to land.May 9 2022, 4:59 PM

lgtm

Can you please add the associated task id in the diff description?
(so one can browse to the related task more easily ;) ?

TIA

I created T4226, I got the request to do that change orally by @bchauvet just after lunch.

Nevertheless, I did not correctly understand the request as the endpoint should not be publicly opened
as its is currently. I will update the diff to enforce that behavior.

Update: Restrict use of the endpoint to privileged users only.

This revision is now accepted and ready to land.May 9 2022, 5:51 PM
anlambert retitled this revision from api/throttling: Lift rate limit for raw endpoint with permission to api/raw: Restrict raw object endpoint use to privileged users.May 9 2022, 5:52 PM
anlambert edited the summary of this revision. (Show Details)
anlambert edited the summary of this revision. (Show Details)

Build is green

Patch application report for D7794 (id=28146)

Rebasing onto b30dc0510f...

Current branch diff-target is up to date.
Changes applied before test
commit 386acfa5904c42b38d0253b9bff1499f2da2e3eb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon May 9 16:22:59 2022 +0200

    api/raw: Restrict raw object endpoint use to privileged users
    
    As that endpoint can leak sensible information regarding authors in
    the archive, it has been decided to not make it publicly available.
    
    So restrict its use to staff users and users with new permission
    "swh.web.api.raw_object".
    
    Also lift rate limit on that new endpoint to ease development
    of the bridge of Software Heritage to IPFS.
    
    Related to T4226

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1808/ for more details.

Why restrict access to contents, directories, and snapshots too?

and actually what sensitive information does this leak about revisions and releases that other endpoints don't?

This revision now requires changes to proceed.May 10 2022, 11:06 AM

Why restrict access to contents, directories, and snapshots too?

I have been asked to restrict the use of that endpoint to specific users by @bchauvet, ping him for more details.

Benoit says the issue is with raw_manifest. So I think we should only require the permission to use the raw_manifest, but still allow everyone to use the other fields.

We'll discuss it with @zack in a bit.

Build is green

Patch application report for D7794 (id=28162)

Rebasing onto b084e4b15e...

Current branch diff-target is up to date.
Changes applied before test
commit 0f8741c1d06a31ada08dc0489a6da40ea7f6d2cf
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon May 9 16:22:59 2022 +0200

    api/raw: Restrict raw object endpoint use to privileged users
    
    As that endpoint can leak sensible information regarding authors in
    the archive, it has been decided to not make it publicly available.
    
    So restrict its use to staff users and users with new permission
    "swh.web.api.raw_object".
    
    Also lift rate limit on that new endpoint to ease development
    of the bridge of Software Heritage to IPFS.
    
    Related to T4226

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1810/ for more details.

vlorentz added a subscriber: douardda.

I just discussed it with @zack and @douardda, and I'm convinced this is fine, now :)

This was a misunderstanding because this endpoint is supposed to return original objects without applying mailmaps, which it currently doesn't. Let's fix it in a future diff

This revision is now accepted and ready to land.May 10 2022, 1:08 PM

I just discussed it with @zack and @douardda, and I'm convinced this is fine, now :)

This was a misunderstanding because this endpoint is supposed to return original objects without applying mailmaps, which it currently doesn't. Let's fix it in a future diff

Oh I see, the ignore_displayname parameter should be used for revision_get and release_get otherwise the hash of the manifest will differ from the object id.

Oh I see, the ignore_displayname parameter should be used for revision_get and release_get otherwise the hash of the manifest will differ from the object id.

yes

Build is green

Patch application report for D7794 (id=28174)

Rebasing onto 84fd757cf8...

Current branch diff-target is up to date.
Changes applied before test
commit 047daa134b44c07bf6cea515cc981c6a971c26bb
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon May 9 16:22:59 2022 +0200

    api/raw: Restrict raw object endpoint use to privileged users
    
    As that endpoint can leak sensible information regarding authors in
    the archive, it has been decided to not make it publicly available.
    
    So restrict its use to staff users and users with new permission
    "swh.web.api.raw_object".
    
    Also lift rate limit on that new endpoint to ease development
    of the bridge of Software Heritage to IPFS.
    
    Related to T4226

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1817/ for more details.

This revision was landed with ongoing or failed builds.May 10 2022, 2:20 PM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D7794 (id=28185)

Rebasing onto 83d38deb16...

Current branch diff-target is up to date.
Changes applied before test
commit 3fc9c9276cff02bb6bd83f36cad0f836fe5ca5f5
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon May 9 16:22:59 2022 +0200

    api/raw: Restrict raw object endpoint use to privileged users
    
    As that endpoint can leak sensible information regarding authors in
    the archive, it has been decided to not make it publicly available.
    
    So restrict its use to staff users and users with new permission
    "swh.web.api.raw_object".
    
    Also lift rate limit on that new endpoint to ease development
    of the bridge of Software Heritage to IPFS.
    
    Related to T4226

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1819/ for more details.