Differential D5859

origin_exists: Handle Internet Archive artifact URLs
ClosedPublic
Actions

Authored by anlambert on Jun 11 2021, 4:43 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete

Commits

rDWAPPSa0db251b3280: common/origin_save/origin_exists: Handle Internet Archive artifact URLs

Summary

Some HTTP hosted tarballs have been archived by the Internet Archive.
In that case to check URL validity and get tarball metadata, HEAD requests
must follow redirection and info regarding last modified date or content
length must be retrieved from different HTTP response headers.

Related to T3365

Diff Detail

Repository

rDWAPPS Web applications

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Jun 11 2021, 4:43 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 11 2021, 4:43 PM

Build is green

Patch application report for D5859 (id=20967)

Rebasing onto da39599f34...

Current branch diff-target is up to date.

Changes applied before test

commit a0db251b3280852ebb078c848f05a07db5bab98f
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jun 11 16:36:55 2021 +0200

    common/origin_save/origin_exists: Handle Internet Archive artifact URLs
    
    Some HTTP hosted tarballs have been archived by the Internet Archive.
    In that case to check URL validity and get tarball metadata, HEAD requests
    must follow redirection and info regarding last modified date or content
    length must be retrieved from different HTTP response headers.
    
    Related to T3365

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/864/ for more details.

Harbormaster completed remote builds in B21947: Diff 20967.Jun 11 2021, 4:51 PM

anlambert requested review of this revision.Jun 11 2021, 4:51 PM