Page MenuHomeSoftware Heritage

debian.loader: Overcome wrong size download information provided
AbandonedPublic

Authored by ardumont on Dec 20 2019, 12:18 PM.

Details

Reviewers
douardda
olasd
Group Reviewers
Reviewers
Summary
  • loader has bad information input (size)
  • We check for download size with that incorrect data.
  • download call function fails because it checks the local size and it's difference

So this opens a flag to bypass that error (only activated for debian). the
loader will still check the provided hashes. If they are also wrong then we
cannot do anything else (from the code standpoint).

That occurrence [1] happens a lot... already 1.1k since i restarted the debian
loader... (less than 1 hour ago or so).

I've checked manually and downloaded files are correct regarding hashes.
Note that it seems to happen mostly for the asc files... not the other ones.

[1] https://sentry.softwareheritage.org/share/issue/49d464bee7b24ad080598b32cd3eb9a8/

[2] https://sentry.softwareheritage.org/organizations/swh/issues/?project=9&query=is%3Aunresolved&sort=freq&statsPeriod=14d

Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 9924
Build 14659: tox-on-jenkinsJenkins
Build 14658: arc lint + arc unit

Event Timeline

vlorentz added inline comments.
swh/loader/package/utils.py
69

A or ((not A) and B) is equivalent to A or B.

Abandonned

  • either adding headers = {'Accept-Encoding': None} to the request instruction and the content-length is fine
  • or computing the length of the response prior to multihashing

(current discussion is on #swh-devel)

swh/loader/package/utils.py
69

doh, right!