Page MenuHomeSoftware Heritage

tarball: Add support for uncompressing crate files
ClosedPublic

Authored by anlambert on Sep 15 2021, 3:55 PM.

Details

Summary

Rust crate files are standard tar.gz archives so add support
to uncompress them through shutil.register_unpack_format.

For the record, those kind of source files archive are processed
by the nixguix loader, see that sentry issue report for instance.

Test Plan

Small crate file has been added in tests data and existing test has been adapted.

Diff Detail

Repository
rDCORE Foundations and core functionalities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6271 (id=22702)

Rebasing onto 632673f6bd...

Current branch diff-target is up to date.
Changes applied before test
commit 590e03756dea42bc00dd25af3c66431bb1d4e5da
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 15 14:20:11 2021 +0200

    tarball: Add support for uncompressing crate files
    
    Rust crate files are standard tar.gz archives so add support
    to uncompress them through shutil.register_unpack_format.
    
    For the record, those kind of source files archive are processed
    by the nixguix loader.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/244/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/core/tests/test_tarball.py
172
This revision is now accepted and ready to land.Sep 15 2021, 4:14 PM
swh/core/tests/test_tarball.py
172
str.endswith(suffix[, start[, end]])

    Return True if the string ends with the specified suffix, otherwise return False. suffix can also be a tuple of suffixes to look for. With optional start, test beginning at that position. With optional end, stop comparing at that position.

Awesome, thanks!

Build is green

Patch application report for D6271 (id=22709)

Rebasing onto 632673f6bd...

Current branch diff-target is up to date.
Changes applied before test
commit 0ef57a4dd346b6c32c1080d38a30922f765c5618
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 15 14:20:11 2021 +0200

    tarball: Add support for uncompressing crate files
    
    Rust crate files are standard tar.gz archives so add support
    to uncompress them through shutil.register_unpack_format.
    
    For the record, those kind of source files archive are processed
    by the nixguix loader.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/245/ for more details.

oh i missed that, great ;)

I pushed rDCOREe268896d7652c7d9ccb83b702b151eb1ef88bd97 after that change which should fix the loading of tgz archives with extension unknown to shutil.unpack_archive.