Page MenuHomeSoftware Heritage

{Cnt|Dir}Loader: Fix standard/nar hash mismatch behavior to fail loading
ClosedPublic

Authored by ardumont on Oct 5 2022, 2:48 PM.

Details

Summary

Prior to this commit, there was a discrepancy between the hash mismatch computations
with "standard" and "nar" computations. This commit fixes the gap between those.

When a hash mismatch occurs, either "nar" or "standard", the issue is caught and the
next mirror url is checked. At the end of it all, if nothing is loaded and errors
exist, this is raised. This fails the visit.

This also adds the missing tests.

Related to T3781
Depends on D8618

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont retitled this revision from {Cnt|Dir}Loader: Add missing tests on hash mismatch failing loading to {Cnt|Dir}Loader: Fix standard/nar hash mismatch behavior to fail loading.Oct 5 2022, 2:51 PM
ardumont edited the summary of this revision. (Show Details)

Build is green

Patch application report for D8630 (id=31153)

Could not rebase; Attempt merge onto b26b98810d...

Updating b26b988..d01f5db
Fast-forward
 swh/loader/core/loader.py            |  86 ++++++++++++++++++---
 swh/loader/core/tests/conftest.py    |  45 +++++++++++
 swh/loader/core/tests/test_loader.py | 143 ++++++++++++++++++++++++++++-------
 swh/loader/core/tests/test_utils.py  |  33 +++++++-
 swh/loader/core/utils.py             |  29 ++++++-
 swh/loader/exception.py              |  14 +++-
 6 files changed, 307 insertions(+), 43 deletions(-)
 create mode 100644 swh/loader/core/tests/conftest.py
Changes applied before test
commit d01f5db27880519f78825c4b2748748acffbffd7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Oct 5 14:10:57 2022 +0200

    {Cnt|Dir}Loader: Add missing tests on hash mismatch failing loading
    
    Prior to this commit, there was a discrepancy between the hash mismatch computations
    with "standard" and "nar" computations. This commit fixes the gap between those.
    
    When a hash mismatch occurs, either "nar" or "standard", the issue is caught and the
    next mirror url is checked. At the end of it all, if nothing is loaded and errors
    exist, this is raised. This fails the visit.
    
    Related to T3781

commit 4d51ad991d3180da683d3f4a9076cd7bb7f54a97
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 19:32:01 2022 +0200

    DirectoryLoader: Check nar hashes when provided
    
    The lister now provides the "checksums_computation". This is either "standard" (for most
    cases as in bare checksums on the object retrieved) or "nar" for some edge case. In that
    case the computation is delegated to the "nix-store" command (which should be present in
    the system running the loading).
    
    This adapts the directory loader to deal with this case.
    
    No work has been done for the ContentLoader yet besides failing the case if a call
    happens with such case.
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/960/ for more details.

Build is green

Patch application report for D8630 (id=31154)

Could not rebase; Attempt merge onto b26b98810d...

Updating b26b988..8aa6dab
Fast-forward
 swh/loader/core/loader.py            |  86 ++++++++++++++++++---
 swh/loader/core/tests/conftest.py    |  45 +++++++++++
 swh/loader/core/tests/test_loader.py | 143 ++++++++++++++++++++++++++++-------
 swh/loader/core/tests/test_utils.py  |  33 +++++++-
 swh/loader/core/utils.py             |  29 ++++++-
 swh/loader/exception.py              |  14 +++-
 6 files changed, 307 insertions(+), 43 deletions(-)
 create mode 100644 swh/loader/core/tests/conftest.py
Changes applied before test
commit 8aa6dab72ad4ab19c71ea010443ab8c49c828c37
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Oct 5 14:10:57 2022 +0200

    {Cnt|Dir}Loader: Fix standard/nar hash mismatch behavior to fail loading
    
    Prior to this commit, there was a discrepancy between the hash mismatch computations
    with "standard" and "nar" computations. This commit fixes the gap between those.
    
    When a hash mismatch occurs, either "nar" or "standard", the issue is caught and the
    next mirror url is checked. At the end of it all, if nothing is loaded and errors
    exist, this is raised. This fails the visit.
    
    This also adds the missing tests.
    
    Related to T3781

commit 4d51ad991d3180da683d3f4a9076cd7bb7f54a97
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 19:32:01 2022 +0200

    DirectoryLoader: Check nar hashes when provided
    
    The lister now provides the "checksums_computation". This is either "standard" (for most
    cases as in bare checksums on the object retrieved) or "nar" for some edge case. In that
    case the computation is delegated to the "nix-store" command (which should be present in
    the system running the loading).
    
    This adapts the directory loader to deal with this case.
    
    No work has been done for the ContentLoader yet besides failing the case if a call
    happens with such case.
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/961/ for more details.

This revision is now accepted and ready to land.Oct 5 2022, 3:13 PM