Page MenuHomeSoftware Heritage

crates: Loader implements incremental mode
Needs RevisionPublic

Authored by franckbret on Tue, Aug 2, 9:31 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Summary

Add incremental support based on sha256 EXTID
Adapt test dataset and add incremental test cases

Related T4104

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
crates-incremental
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30576
Build 47804: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47803: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8171 (id=29506)

Rebasing onto 01547b8edf...

Current branch diff-target is up to date.
Changes applied before test
commit c41147833da8e2353721b071c3145d36d2fd7955
Author: Franck Bret <franck.bret@octobus.net>
Date:   Tue Aug 2 09:21:24 2022 +0200

    crates: Loader implements incremental mode
    
    Add incremental support based on sha256 EXTID
    Adapt test dataset and add incremental test cases
    
    Related T4104

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/813/ for more details.

@vlorentz @ardumont Here is a proposal to make the crates loader incremental.

Hello, was there a discussion leading to this implementation? If so, it'd be nice to mention it in the description (please).

vlorentz added inline comments.
swh/loader/package/crates/loader.py
167

Are all attributes of CratesPackageInfo guaranteed to be generated deterministically from content covered by this hash?

(See D8173 for details)

306–308

the first assertion is redundant; artifact["checksums"]["sha256"] would raise KeyError anyway

swh/loader/package/crates/tests/test_crates.py
427–429

Create a new loader instance, so tests behave more like production

This revision now requires changes to proceed.Wed, Aug 3, 12:58 PM