Page MenuHomeSoftware Heritage
Differential D7192

Route for fetching Git-encoded objects
ClosedPublic
Actions

Authored by Ericson2314 on Feb 16 2022, 5:50 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDWAPPS25a999821b3f: Adds an API route for fetching objects as Git objects
rDWAPPS43b197d8357f: Deduplicate `SWHID_RE`
rDWAPPS9cd95df4775c: Extend test utits with new functions
Required Signatures
L3 Software Heritage Contributor License Agreement, version 1.0
Summary

Adds an API route for fetching objects as Git objects

This route is needed for the implementation of the SWH Archive-IPFS
bridge. Specifically, we need a way to, given a SWHID, fetch the exact
data that was used to hash an object.

Diff Detail

Repository
rDWAPPS Web applications
Branch
arcpatch-D7192
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28867
Build 45118: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 45117: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
182 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.api.views.test_raw::Tests / Python tests / test_api_raw_directory
api_client = <rest_framework.test.APIClient object at 0x7f4ba8ebacc0> archive_data = <swh.web.tests.conftest._ArchiveData object at 0x7f4ba8c1c5c0> directory = '3688c09fb017133403e42b4542a339c6a551f2ae'
432 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.add_forge_now.test_migration::Tests / Python tests / test_add_forge_now_allow_no_comment
601 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.add_forge_now.test_migration::Tests / Python tests / test_add_forge_now_initial_migration
426 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.add_forge_now.test_migration::Tests / Python tests / test_add_forge_now_store_submitter_forward_username
187 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.add_forge_now.test_models::Tests / Python tests / test_allowed_next_statuses[RequestStatus.PENDING-allowed_next_statuses0]
View Full Test Results (1 Failed · 964 Passed · 5 Skipped)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Harbormaster failed remote builds in B28930: Diff 27869!Apr 28 2022, 12:35 AM
Ericson2314 updated this revision to Diff 27870.Apr 28 2022, 12:37 AM

Try to fix without with_data

swh-public-ci added a comment.Apr 28 2022, 12:56 AM

Build is green

Patch application report for D7192 (id=27870)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit db6856571264cdc14ced37ad895643b1d8435ec5
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html
    
    Differential Revision: https://forge.softwareheritage.org/D7192

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1755/ for more details.

Harbormaster completed remote builds in B28932: Diff 27870.Apr 28 2022, 12:56 AM
Ericson2314 updated this revision to Diff 27871.Apr 28 2022, 12:56 AM

It works! Clean up commit history and stop filtering the tests

swh-public-ci added a comment.Apr 28 2022, 1:17 AM

Build is green

Patch application report for D7192 (id=27871)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit 71c3e064a272a6f1e5a4a1f0668930e26684216f
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 241f90ba38c26924aef64b62e2d318a51d4df274
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend `swh.web.tests.data` with new functions
    
    - `random_sha1_bytes`
    
    - `random_object_type`
    
    - `random_swhid`
    
    The first two are used to make the third, but useful in their own right.
    The third will be used in the following commit.

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1756/ for more details.

Harbormaster completed remote builds in B28933: Diff 27871.Apr 28 2022, 1:17 AM
Ericson2314 marked 2 inline comments as done.Apr 28 2022, 6:49 AM
vlorentz added a comment.Apr 28 2022, 7:50 AM

Looks good to me, thanks.

swh/web/api/views/raw.py
41

This ref won't work because it's not the name of an existing function. Use this instead:

see ``*_git_object`` functions in :mod:`swh.model.git_objects`
49–51

don't invalid SWHIDs raise a 404 too?

swh/web/tests/api/views/test_raw.py
17–25

Could you make this test run for all SWHID types?

Ericson2314 updated this revision to Diff 27890.Apr 28 2022, 5:11 PM

Address some review comments

swh-public-ci added a comment.Apr 28 2022, 5:13 PM

Build has FAILED

Patch application report for D7192 (id=27890)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit e87641d4ab986ef89c646cbc00756aea0f9bd3da
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 40b0fa4dfa7b571b3374a344273d7b9db83a0042
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
      - `random_object_type`
      - `random_swhid`
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a fixture

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1758/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1758/console

Harbormaster failed remote builds in B28954: Diff 27890!Apr 28 2022, 5:13 PM
Ericson2314 marked 2 inline comments as done.Apr 28 2022, 5:24 PM
Ericson2314 added inline comments.
swh/web/api/views/raw.py
49–51

I have no idea :). This was inherited from the snapshot handler the patch as originally based on. I am just hoping the thrown exceptions do that! :)

swh/web/tests/api/views/test_raw.py
17–25

I have made a fixture that does this.

Ericson2314 updated this revision to Diff 27891.Apr 28 2022, 5:25 PM
Ericson2314 marked an inline comment as done.

Remove unused imports

swh-public-ci added a comment.Apr 28 2022, 5:44 PM

Build has FAILED

Patch application report for D7192 (id=27891)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit 291e668fb9bb863936ec2e8d85e12ab6a86f25f9
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 40b0fa4dfa7b571b3374a344273d7b9db83a0042
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
      - `random_object_type`
      - `random_swhid`
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a fixture

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1759/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1759/console

Harbormaster failed remote builds in B28955: Diff 27891!Apr 28 2022, 5:45 PM
Ericson2314 updated this revision to Diff 27894.Apr 28 2022, 6:26 PM

It would seem I did a no-op revise before

swh-public-ci added a comment.Apr 28 2022, 6:44 PM

Build is green

Patch application report for D7192 (id=27894)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit b70379b50ba8ed6a76dc62dd45c6194f1398f142
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 725942c027d8010e587a7e15908002c88fcc225e
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
      - `random_object_type`
      - `random_swhid`
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a fixture

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1760/ for more details.

Harbormaster completed remote builds in B28959: Diff 27894.Apr 28 2022, 6:44 PM
vlorentz added inline comments.Apr 28 2022, 6:52 PM
swh/web/api/views/raw.py
49–51

I don't see anything in the code that would raise a 400; so it's either a 404 (if Django rejects based on the regexp) or a 500.

swh/web/tests/data.py
78–86 ↗(On Diff #27894)

this is dead code now

Ericson2314 updated this revision to Diff 27895.Apr 28 2022, 7:31 PM

Remove dead code, remove 400 from docs

Ericson2314 marked 3 inline comments as done.Apr 28 2022, 7:32 PM
Ericson2314 added inline comments.
swh/web/api/views/raw.py
49–51

What happens if CoreSWHID.from_string throws a ValidationError?

Still, regardless you are right that even if that is a 400, regex non-matches will be a 404.

One of the swh.web.view.vault one taking a SWHID mentions the 404, but the others don't so I will just drop it.

swh/web/tests/data.py
78–86 ↗(On Diff #27894)

Sure. I misinterpreted you before thinking it would be good ot have anyways. Will remove.

vlorentz added a comment.Apr 28 2022, 7:44 PM

LGTM.

@anlambert Good for you too?

swh/web/api/views/raw.py
49–51

it raises a 500 because it won't be caught (afaik).

ack

swh-public-ci added a comment.Apr 28 2022, 7:58 PM

Build was aborted

Patch application report for D7192 (id=27895)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit b5c42d617dafc6562a7e31f416fcfe3f8506e35d
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit e48569769b9468dcdd54ba5eb73045a642c5291f
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1761/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1761/console

Harbormaster failed remote builds in B28960: Diff 27895!Apr 28 2022, 7:58 PM
Ericson2314 marked 2 inline comments as done.Apr 28 2022, 8:24 PM

I'll wait for now pending @anlambert review, but let me know if I should push again to to trigger a new build after the last one seems to have died spuriously (if I am not misunderstanding).

Ericson2314 updated this revision to Diff 27896.Apr 29 2022, 6:15 AM

Push same stuff again to restart job

Figure we're about to be asleep in both our timezones, so might as well :).

swh-public-ci added a comment.Apr 29 2022, 6:42 AM

Build was aborted

Patch application report for D7192 (id=27896)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit 51e99a5707e01f2d6b800a04ac2f85e858199dbd
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit e48569769b9468dcdd54ba5eb73045a642c5291f
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1762/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1762/console

Harbormaster failed remote builds in B28961: Diff 27896!Apr 29 2022, 6:42 AM
Ericson2314 added a comment.Apr 29 2022, 5:50 PM

Do you all know what is causing these CI failures?

Ericson2314 updated this revision to Diff 27919.May 1 2022, 6:31 PM

Kick off CI again after I saw swh-model's wor

vlorentz added a comment.May 1 2022, 6:35 PM
In D7192#201132, @Ericson2314 wrote:

Do you all know what is causing these CI failures?

just Cypress tests timing out as usual

Ericson2314 added a comment.May 1 2022, 6:44 PM
In D7192#201166, @vlorentz wrote:
In D7192#201132, @Ericson2314 wrote:

Do you all know what is causing these CI failures?

just Cypress tests timing out as usual

OK thanks, well, hopefully we can sort it out and get the second review French business hours next week :)

I've made drafts of a) the IPFS plugin using this new route b) snapshot encoding and decoding on the IPFS side, so once I can test those this we should be close to finished!

vlorentz added a comment.May 1 2022, 6:56 PM

Thanks. Two things I forgot below:

swh/web/api/views/raw.py
1

should be 2022 instead

86–91

This usually does not matter, but some directories' git_object cannot be entirely rebuilt just from the list of entries for various reasons.

(and please add a test for it; there is an example here: https://forge.softwareheritage.org/source/swh-vault/browse/master/swh/vault/tests/test_cookers.py$1121-1141 )

swh/web/tests/api/views/test_raw.py
1

ditto

swh/web/tests/conftest.py
1 ↗(On Diff #27919)

and you should add 2022 here

swh-public-ci added a comment.May 1 2022, 6:58 PM

Build was aborted

Patch application report for D7192 (id=27919)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit 51e99a5707e01f2d6b800a04ac2f85e858199dbd
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit e48569769b9468dcdd54ba5eb73045a642c5291f
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1763/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1763/console

Harbormaster failed remote builds in B28984: Diff 27919!May 1 2022, 6:58 PM
Ericson2314 updated this revision to Diff 27920.May 1 2022, 7:15 PM
Ericson2314 marked 3 inline comments as done.

Fix copyright dates

Ericson2314 added a comment.May 1 2022, 7:16 PM

(Mark notes to fix copyright as done)

Ericson2314 updated this revision to Diff 27921.May 1 2022, 7:20 PM

Include raw manifest in directory object

vlorentz added inline comments.May 1 2022, 7:28 PM
swh/web/api/views/raw.py
86–91

oh my bad, it should be [object_id] instead of [0] (the method returns a dict for some reason)

swh-public-ci added a comment.May 1 2022, 7:42 PM

Build was aborted

Patch application report for D7192 (id=27920)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit fe12bd286e8ce563205d41a0d3b154a6e5d766c8
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 7277f4d8f46296003a779245448fb48fb89a4b2a
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1764/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1764/console

Harbormaster failed remote builds in B28985: Diff 27920!May 1 2022, 7:42 PM
swh-public-ci added a comment.May 1 2022, 7:45 PM

Build has FAILED

Patch application report for D7192 (id=27921)

Rebasing onto 25bf6ca843...

Current branch diff-target is up to date.
Changes applied before test
commit b2f55a240c44b40768de0cffa759ce40f019a0c7
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 7277f4d8f46296003a779245448fb48fb89a4b2a
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit b5f20ca9ddfcc2745a76a715e4c892c19c0d0942
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1765/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1765/console

Harbormaster failed remote builds in B28986: Diff 27921!May 1 2022, 7:45 PM
Ericson2314 added inline comments.May 1 2022, 8:08 PM
swh/web/api/views/raw.py
86–91

I this makes me think directory reassembly is subtle enough that it deserves its own function. I therefore opened D7720 to make one analogous to the snapshot one.

We could still do an integration test for the web interface, but with this division of labor directory_get_all_entries and directory_git_object can also be tested in isolation.

vlorentz added a comment.May 3 2022, 1:24 PM

D7722 landed and released, you can now use that new storage.algos.directory.directory_get() function.

Ericson2314 updated this revision to Diff 27953.May 3 2022, 4:25 PM

Use new swh.storage.algos.directory.directory_get

Rebase on lastest master, where cypress CI issue is hopefully fixed!

swh-public-ci added a comment.May 3 2022, 4:27 PM

Build has FAILED

Patch application report for D7192 (id=27953)

Rebasing onto 4a6bc69b2f...

Current branch diff-target is up to date.
Changes applied before test
commit 6de4c34b57751fa0284644dbae940051fce2a7bf
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 055741dd3b7d340833e862951f26407fbe3d2497
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit a4d06a29323a1d26061e23dfa126d983b2cdc955
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1773/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1773/console

Harbormaster failed remote builds in B29031: Diff 27953!May 3 2022, 4:27 PM
swh-public-ci added a comment.May 3 2022, 4:54 PM

Build is green

Patch application report for D7192 (id=27953)

Rebasing onto 4a6bc69b2f...

Current branch diff-target is up to date.
Changes applied before test
commit 6de4c34b57751fa0284644dbae940051fce2a7bf
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 055741dd3b7d340833e862951f26407fbe3d2497
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit a4d06a29323a1d26061e23dfa126d983b2cdc955
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1775/ for more details.

Harbormaster completed remote builds in B29031: Diff 27953.May 3 2022, 4:54 PM
vlorentz accepted this revision.May 3 2022, 4:57 PM
This revision is now accepted and ready to land.May 3 2022, 4:57 PM
Ericson2314 updated this revision to Diff 27962.May 3 2022, 7:56 PM

Rebase as master changed since

Ericson2314 added a comment.May 3 2022, 7:57 PM

@vlorentz is anyone else (@anlambert?) going to review, or should I push this once it passes again?

swh-public-ci added a comment.May 3 2022, 8:16 PM

Build is green

Patch application report for D7192 (id=27962)

Rebasing onto d9fb8522ed...

Current branch diff-target is up to date.
Changes applied before test
commit 25a999821b3fe680f7a9c7138c058a102b807c3a
Author: Amélia Liao <me@amelia.how>
Date:   Mon Apr 25 19:56:26 2022 -0400

    Adds an API route for fetching objects as Git objects
    
    This route is needed for the implementation of the SWH Archive-IPFS
    bridge. Specifically, we need a way to, given a SWHID, fetch the exact
    data that was used to hash an object.
    
    The design follows the plan @vlorentz put forth in
    https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00042.html

commit 9cd95df4775c9ba0de0e0465654c6489b96f9c77
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:48:35 2022 -0400

    Extend test utits with new functions
    
    In `swh.web.tests.data`:
    
      - `random_sha1_bytes
    
    In `swh.web.tests.conftest`:
    
      - `unknown_core_swhid`, a pytest "fixture"

commit 43b197d8357ffbdcd21afe393bb4c2d9fcd94a16
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date:   Wed Apr 27 18:46:37 2022 -0400

    Deduplicate `SWHID_RE`
    
    Put it in `swh.web.common.utils` and import from there.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1778/ for more details.

Harbormaster completed remote builds in B29040: Diff 27962.May 3 2022, 8:16 PM
Closed by commit rDWAPPS43b197d8357f: Deduplicate `SWHID_RE` (authored by Ericson2314). · Explain WhyMay 3 2022, 10:08 PM
This revision was automatically updated to reflect the committed changes.
Ericson2314 added a commit: rDWAPPS43b197d8357f: Deduplicate `SWHID_RE`.
Ericson2314 added a commit: rDWAPPS9cd95df4775c: Extend test utits with new functions.
Ericson2314 added a commit: rDWAPPS25a999821b3f: Adds an API route for fetching objects as Git objects.
Ericson2314 added a comment.May 3 2022, 10:09 PM

Eh, I'll just assume, given the timezone latency, that going down to one review was intentional. (With the directory algorithm factored out into swh-storage, this patch is now simpler and more "just do the obvious thing" too.)

Ericson2314 added a comment.May 3 2022, 10:11 PM

(Post merge, the diff that shows up is just the first commit, but they all are on master.)

Ericson2314 added a comment.May 4 2022, 12:20 AM

Ah. and I guess there needs to be another round of tagging and whatnot to trigger a website redeploy?

Revision Contents
Changeset List

PathSize
swh/
web/
api/
views/
90 lines
common/
14 lines
tests/
api/
views/
62 lines

Diff 27807

View Options

swh/web/api/views/raw.py

Loading...
View Options

swh/web/common/archive.py

Loading...
View Options

swh/web/tests/api/views/test_raw.py

Loading...
About · Developer information · Legal