Diffusion Web applications a0db251b3280

common/origin_save/origin_exists: Handle Internet Archive artifact URLs
a0db251b3280
Actions

Tags

None

Subscribers

None

Description

common/origin_save/origin_exists: Handle Internet Archive artifact URLs

Some HTTP hosted tarballs have been archived by the Internet Archive.
In that case to check URL validity and get tarball metadata, HEAD requests
must follow redirection and info regarding last modified date or content
length must be retrieved from different HTTP response headers.

Related to T3365

Details

Provenance

anlambert	Authored on Jun 11 2021, 4:36 PM
anlambert	Pushed on Jun 11 2021, 5:23 PM

Differential Revision

D5859: common/origin_save/origin_exists: Handle Internet Archive artifact URLs

Parents

rDWAPPSda39599f3484: mypy: Fix errors with v0.900 release

Branches

Unknown

Tags

Unknown

References

Tasks

T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete

Build Status

Buildable 21949
Build 34140: test-and-build	Jenkins console · Jenkins

Event Timeline

anlambert committed rDWAPPSa0db251b3280: common/origin_save/origin_exists: Handle Internet Archive artifact URLs (authored by anlambert).Jun 11 2021, 4:37 PM

anlambert added a task: T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.Jun 11 2021, 5:23 PM

anlambert added an edge: D5859: common/origin_save/origin_exists: Handle Internet Archive artifact URLs.

Harbormaster completed building B21949: rDWAPPSa0db251b3280: common/origin_save/origin_exists: Handle Internet Archive artifact URLs.Jun 11 2021, 5:31 PM

swh-public-ci mentioned this in D5828: Redirect to the browse origin page when the user searchs for an exact, already archived URL.Jun 14 2021, 3:40 PM

swh-public-ci mentioned this in D5866: assets: Simplify promises handling with js async/await syntax.Jun 14 2021, 6:10 PM

swh-public-ci mentioned this in D5858: Schedule save code now as recurring origins to ingest when successful.Jun 15 2021, 11:39 AM

Changes (2)

Path

Size

swh/

web/

common/

tests/

common/

test_origin_save.py

rDWAPPSa0db251b3280

swh/web/common/origin_save.py

Loading...

swh/web/tests/common/test_origin_save.py

Loading...