Page MenuHomeSoftware Heritage

Add support for more tarball/zip extensions
ClosedPublic

Authored by ardumont on Oct 24 2022, 3:53 PM.

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8757 (id=31571)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 07648eda68ea4b2e394612501a8aa2b912a3c567
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball/zip extensions
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/812/ for more details.

Build is green

Patch application report for D8757 (id=31573)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 44b1edfc96c96e4074661b648b5e0108c5da3618
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/813/ for more details.

anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/lister/nixguix/tests/test_lister.py
214

real tarball artifact seems a better comment imho

This revision is now accepted and ready to land.Oct 24 2022, 4:16 PM
swh/lister/nixguix/tests/test_lister.py
214

right!

fwiw, i'm looking at adding support to all those new zip archives.
I'm actually checking our core tarball module already supports those (they do so far).
It's just not wired.

Build is green

Patch application report for D8757 (id=31574)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 80150438e0b4dedb21dd8938f23504d69c6144b9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/814/ for more details.

swh/lister/nixguix/tests/test_lister.py
214

yes the tarball module use mime type as last resort so it should detect zip files with different extensions.

swh/lister/nixguix/tests/test_lister.py
214

You seem to imply that [1] is actually not needed, do you concur?

That'd be nice.

[1] D8758

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/lister/__init__.py
54

missing comma

This revision now requires changes to proceed.Oct 24 2022, 4:39 PM
This revision now requires changes to proceed.Oct 24 2022, 4:40 PM

Build is green

Patch application report for D8757 (id=31577)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit fef2e46fa17056b5e01399b719afad4368849463
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/815/ for more details.

Build is green

Patch application report for D8757 (id=31578)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit b7c6e1e1a74282b68949553eb4bfd1e365f3cfdb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/816/ for more details.

swh/lister/nixguix/tests/test_lister.py
214

You can keep the exception for war files I think but you can remove the other ones yes.

ardumont added inline comments.
swh/lister/nixguix/tests/test_lister.py
214

yes, thx done in the other diff ;)

This revision is now accepted and ready to land.Oct 24 2022, 6:26 PM

Add dataset around newly supported tarball formats

Build is green

Patch application report for D8757 (id=31585)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 7b14e6171c7c90fa4b6d2c22f20d852bf2734fca
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/817/ for more details.

Add missing new archive format in test manifest

Build is green

Patch application report for D8757 (id=31586)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 31eb5f637f69cc0e500a0ba8a82dc740ed63a1e3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/818/ for more details.