Page MenuHomeSoftware Heritage
Differential D8757

Add support for more tarball/zip extensions
ClosedPublic
Actions

Authored by ardumont on Oct 24 2022, 3:53 PM.

Details

Reviewers
anlambert
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3781: Replace the Nixguix loader with a lister
Commits
rDLS31eb5f637f69: Add support for more tarball recognition based on extensions
Summary

Related to T3781
Related to P1503

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32531
Build 50951: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50950: arc lint + arc unit

Event Timeline

ardumont created this revision.Oct 24 2022, 3:53 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 24 2022, 3:53 PM
swh-public-ci added a comment.Oct 24 2022, 3:59 PM

Build is green

Patch application report for D8757 (id=31571)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 07648eda68ea4b2e394612501a8aa2b912a3c567
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball/zip extensions
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/812/ for more details.

Harbormaster completed remote builds in B32525: Diff 31571.Oct 24 2022, 3:59 PM
ardumont requested review of this revision.Oct 24 2022, 3:59 PM
ardumont updated this revision to Diff 31573.Oct 24 2022, 4:09 PM

Update test too

swh-public-ci added a comment.Oct 24 2022, 4:15 PM

Build is green

Patch application report for D8757 (id=31573)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 44b1edfc96c96e4074661b648b5e0108c5da3618
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/813/ for more details.

Harbormaster completed remote builds in B32527: Diff 31573.Oct 24 2022, 4:15 PM
anlambert accepted this revision.Oct 24 2022, 4:16 PM
anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/lister/nixguix/tests/test_lister.py
214

real tarball artifact seems a better comment imho

This revision is now accepted and ready to land.Oct 24 2022, 4:16 PM
ardumont added inline comments.Oct 24 2022, 4:26 PM
swh/lister/nixguix/tests/test_lister.py
214

right!

fwiw, i'm looking at adding support to all those new zip archives.
I'm actually checking our core tarball module already supports those (they do so far).
It's just not wired.

ardumont updated this revision to Diff 31574.Oct 24 2022, 4:28 PM

Fix test comment

swh-public-ci added a comment.Oct 24 2022, 4:33 PM

Build is green

Patch application report for D8757 (id=31574)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 80150438e0b4dedb21dd8938f23504d69c6144b9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/814/ for more details.

Harbormaster completed remote builds in B32528: Diff 31574.Oct 24 2022, 4:33 PM
anlambert added inline comments.Oct 24 2022, 4:34 PM
swh/lister/nixguix/tests/test_lister.py
214

yes the tarball module use mime type as last resort so it should detect zip files with different extensions.

ardumont added inline comments.Oct 24 2022, 4:39 PM
swh/lister/nixguix/tests/test_lister.py
214

You seem to imply that [1] is actually not needed, do you concur?

That'd be nice.

[1] D8758

vlorentz requested changes to this revision.Oct 24 2022, 4:39 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/lister/__init__.py
54

missing comma

This revision now requires changes to proceed.Oct 24 2022, 4:39 PM
ardumont updated this revision to Diff 31577.Oct 24 2022, 4:40 PM

Fix typos

vlorentz requested changes to this revision.Oct 24 2022, 4:40 PM
This revision now requires changes to proceed.Oct 24 2022, 4:40 PM
ardumont updated this revision to Diff 31578.Oct 24 2022, 4:41 PM

Fix missing comma

ardumont marked 3 inline comments as done.Oct 24 2022, 4:43 PM
swh-public-ci added a comment.Oct 24 2022, 4:46 PM

Build is green

Patch application report for D8757 (id=31577)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit fef2e46fa17056b5e01399b719afad4368849463
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/815/ for more details.

Harbormaster completed remote builds in B32531: Diff 31577.Oct 24 2022, 4:46 PM
swh-public-ci added a comment.Oct 24 2022, 4:47 PM

Build is green

Patch application report for D8757 (id=31578)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit b7c6e1e1a74282b68949553eb4bfd1e365f3cfdb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/816/ for more details.

Harbormaster completed remote builds in B32532: Diff 31578.Oct 24 2022, 4:47 PM
anlambert added inline comments.Oct 24 2022, 4:47 PM
swh/lister/nixguix/tests/test_lister.py
214

You can keep the exception for war files I think but you can remove the other ones yes.

ardumont marked an inline comment as done.Oct 24 2022, 5:19 PM
ardumont added inline comments.
swh/lister/nixguix/tests/test_lister.py
214

yes, thx done in the other diff ;)

vlorentz accepted this revision.Oct 24 2022, 6:26 PM
This revision is now accepted and ready to land.Oct 24 2022, 6:26 PM
ardumont marked an inline comment as done.Oct 25 2022, 9:31 AM
ardumont updated this revision to Diff 31585.Oct 25 2022, 9:49 AM

Add dataset around newly supported tarball formats

swh-public-ci added a comment.Oct 25 2022, 9:55 AM

Build is green

Patch application report for D8757 (id=31585)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 7b14e6171c7c90fa4b6d2c22f20d852bf2734fca
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/817/ for more details.

Harbormaster completed remote builds in B32539: Diff 31585.Oct 25 2022, 9:55 AM
ardumont updated this revision to Diff 31586.EditedOct 25 2022, 10:06 AM

Add missing new archive format in test manifest

ardumont added a child revision: D8761: nixguix/test: Add all supported tarball extensions to test manifest.Oct 25 2022, 10:09 AM
swh-public-ci added a comment.Oct 25 2022, 10:12 AM

Build is green

Patch application report for D8757 (id=31586)

Rebasing onto 8a82bbf95f...

Current branch diff-target is up to date.
Changes applied before test
commit 31eb5f637f69cc0e500a0ba8a82dc740ed63a1e3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 24 15:44:47 2022 +0200

    Add support for more tarball recognition based on extensions
    
    This requires to open those extensions to be supported by loaders too (in
    swh.core.tarball).
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/818/ for more details.

Harbormaster completed remote builds in B32540: Diff 31586.Oct 25 2022, 10:12 AM
Closed by commit rDLS31eb5f637f69: Add support for more tarball recognition based on extensions (authored by ardumont). · Explain WhyOct 25 2022, 10:16 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS31eb5f637f69: Add support for more tarball recognition based on extensions.
ardumont mentioned this in T3781: Replace the Nixguix loader with a lister.Oct 25 2022, 2:50 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
lister/
6 lines81%
Loading...
nixguix/
tests/
data/
9 lines--
2 lines100%
Loading...

Diff 31577

View Options

swh/lister/__init__.py

Loading...
View Options

swh/lister/nixguix/tests/data/sources-success.json

Loading...
View Options

swh/lister/nixguix/tests/test_lister.py

Loading...
About · Developer information · Legal