Page MenuHomeSoftware Heritage

[WIP] archive.loader: Allow archive loader to deal with nixguix archives
AbandonedPublic

Authored by ardumont on Sep 6 2022, 6:21 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T3781: Replace the Nixguix loader with a lister
Summary
  • add tests -> Those fails [1]

- [ ] Figure out how to deal with missing version which is currently a requirement in the archive loader implementation.

We cannot adapt the current ArchiveLoader[PackageLoader]. The actual implementation requires versions and the algo is
based on that versions list we do not have. We may have at some point for some case but we won't necessarily have it for all.

We need new ContentLoader[BaseLoader] and DirectoryLoader[BaseLoader] implementations to deal with such tarballs individually.

Related to T3781

[1] D8406#222815

Diff Detail

Event Timeline

Build has FAILED

Patch application report for D8406 (id=30337)

Rebasing onto 6cdf6d3004...

First, rewinding head to replay your work on top of it...
Applying: archive.loader: Allow archive loader to deal with nixguix archives
Changes applied before test
commit 33ddd675083332ebe952fd80f753e13fd2d74a91
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Sep 6 18:20:17 2022 +0200

    archive.loader: Allow archive loader to deal with nixguix archives
    
    Related to T3781

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/860/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/860/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 6 2022, 6:22 PM
Harbormaster failed remote builds in B31364: Diff 30337!

Fix tests
I still need to add some scenario for the new integrity archives though

Build has FAILED

Patch application report for D8406 (id=30338)

Rebasing onto 6cdf6d3004...

Current branch diff-target is up to date.
Changes applied before test
commit 82833c5dba3810c21dfc46cb7d823c2d0f2ba438
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Sep 6 18:20:17 2022 +0200

    archive.loader: Allow archive loader to deal with nixguix archives
    
    Related to T3781

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/861/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/861/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 6 2022, 6:45 PM
Harbormaster failed remote builds in B31365: Diff 30338!

Build is green

Patch application report for D8406 (id=30339)

Rebasing onto 6cdf6d3004...

Current branch diff-target is up to date.
Changes applied before test
commit f4db8ab05f56df393238b61bb7d6c87f3668cd82
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Sep 6 18:20:17 2022 +0200

    archive.loader: Allow archive loader to deal with nixguix archives
    
    Related to T3781

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/862/ for more details.

vlorentz added inline comments.
swh/loader/package/archive/loader.py
35

add a link to the format's spec: https://www.w3.org/TR/SRI/#the-integrity-attribute and mention it should follow the hash-expression rule

37–41

Make it clear the one with a checksum is prefered

vlorentz retitled this revision from wip: archive.loader: Allow archive loader to deal with nixguix archives to [WIP] archive.loader: Allow archive loader to deal with nixguix archives.Sep 15 2022, 10:40 AM

Add tests that explicitly fail as it's missing the required version in the current
package loader implementation.

This makes this diff non-landable and makes new dedicated ContentLoader[BaseLoader] and
DirectoryLoader[BaseLoader] required to be implemented in the refactoring nixguix
listing paths.

ardumont edited the summary of this revision. (Show Details)

Build has FAILED

Patch application report for D8406 (id=30938)

Rebasing onto 7375a83c2f...

Current branch diff-target is up to date.
Changes applied before test
commit 6cbbe53e736755de31fd5bdf24520467caa86c33
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Sep 9 11:31:45 2022 +0200

    wip: Add test on integrity tarball ingestion
    
    That fails as package loader actually expects versions but integrity package are only
    describing standalone tarball with only one integrity field.

commit e89153fa2f6f3cf09c6bc67abbd9446f6350cb70
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Sep 6 18:20:17 2022 +0200

    archive.loader: Allow archive loader to deal with nixguix archives
    
    Related to T3781

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/911/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/911/console

ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

This cannot work as explained time and again.