Page MenuHomeSoftware Heritage

DirectoryLoader: Check nar hashes when provided
ClosedPublic

Authored by ardumont on Oct 4 2022, 7:33 PM.

Details

Summary

The lister now provides the "checksums_computation". This is either "standard" (for most
cases as in bare checksums on the object retrieved) or "nar" for some edge case. In that
case the computation is delegated to the "nix-store" command (which should be present in
the system running the loading).

This adapts the directory loader to deal with this case.

No work has been done for the ContentLoader yet besides failing the case if a call
happens with such case.

Related to T3781
Depends on D8627

Diff Detail

Event Timeline

Build is green

Patch application report for D8618 (id=31120)

Rebasing onto b26b98810d...

Current branch diff-target is up to date.
Changes applied before test
commit f8c035cd8b48ca938a2f3fd4f9d53aeea2489abb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 19:32:01 2022 +0200

    {Cnt|Dir}Loader: Check nar hashes when nix-store is available
    
    The lister nows provides the "checksums_computation" which is either "standard" (for
    most cases) and "nar" for some edge case.
    
    This adapts the content and directory loader to deal with those.
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/956/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/core/loader.py
679

You need to add this to the docstring, along with the meaning of each value.

857–877

pathlib ftw :)

swh/loader/core/tests/test_loader.py
805

Please add it to Jenkins before landing this diff

swh/loader/core/utils.py
134

To make the above change work

This revision now requires changes to proceed.Oct 5 2022, 10:39 AM
swh/loader/core/tests/test_loader.py
805

that's more complicated than this, i think jenkins runs buster?

ardumont added a subscriber: anlambert.
ardumont added inline comments.
swh/loader/core/loader.py
760

I still need to check this happens for real for file...

swh/loader/core/tests/test_loader.py
805

@anlambert updated jenkins with D8627 \o/

ardumont added inline comments.
swh/loader/core/loader.py
760

I'm gonna drop it for now and run with an assertion on checksum_computations == standard and see what happens in docker.

ardumont added inline comments.
swh/loader/core/loader.py
857–877

\m/

ardumont retitled this revision from {Cnt|Dir}Loader: Check nar hashes when nix-store is available to DirectoryLoader: Check nar hashes when provided.Oct 5 2022, 12:57 PM
ardumont edited the summary of this revision. (Show Details)
ardumont marked an inline comment as done.

Adapt according to review and add missing coverage

Build is green

Patch application report for D8618 (id=31146)

Rebasing onto b26b98810d...

Current branch diff-target is up to date.
Changes applied before test
commit 4d51ad991d3180da683d3f4a9076cd7bb7f54a97
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 4 19:32:01 2022 +0200

    DirectoryLoader: Check nar hashes when provided
    
    The lister now provides the "checksums_computation". This is either "standard" (for most
    cases as in bare checksums on the object retrieved) or "nar" for some edge case. In that
    case the computation is delegated to the "nix-store" command (which should be present in
    the system running the loading).
    
    This adapts the directory loader to deal with this case.
    
    No work has been done for the ContentLoader yet besides failing the case if a call
    happens with such case.
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/959/ for more details.

This revision is now accepted and ready to land.Oct 5 2022, 1:08 PM