Page MenuHomeSoftware Heritage

maven: Add support for md5 checkums to check download integrity
ClosedPublic

Authored by anlambert on Nov 14 2022, 4:26 PM.

Details

Summary

Some maven artifacts do not have any sha1 sums computed but rather md5
ones so handle these edge cases to still check download integrity of
jar files.

See https://repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-test-annotations-common/1.5.30-M1/ for instance.

Depends on D8839

Fixes SWH-LOADER-CORE-1BP.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8840 (id=31854)

Could not rebase; Attempt merge onto bf2cb039d5...

Updating bf2cb03..301502c
Fast-forward
 swh/loader/package/maven/loader.py                 |  11 ++-
 .../sprova4j-0.1.0-sources.jar.sha1                |   1 -
 ...aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar} | Bin
 ...i_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar.md5 |   1 +
 ...ven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0.pom} |   0
 ...aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar} | Bin
 ...sprova4j_0.1.1_sprova4j-0.1.1-sources.jar.sha1} |   0
 ...ven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1.pom} |   0
 swh/loader/package/maven/tests/test_maven.py       | 107 +++++++--------------
 9 files changed, 43 insertions(+), 77 deletions(-)
 delete mode 100644 swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.0-sources.jar.sha1
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.0-sources.jar => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar} (100%)
 create mode 100644 swh/loader/package/maven/tests/data/https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar.md5
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.0.pom => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0.pom} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1-sources.jar => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1-sources.jar.sha1 => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar.sha1} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1.pom => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1.pom} (100%)
Changes applied before test
commit 301502cbf8dc5d69c01e86278161628b157c9c47
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 14 16:19:27 2022 +0100

    maven: Add support for md5 checkums to check download integrity
    
    Some maven artifacts do not have any sha1 sums computed but rather md5
    ones so handle these edge cases to still check download integrity of
    jar files.

commit 5778cfce008af543c566197f0c229637bceac287
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 14 16:12:12 2022 +0100

    maven: Simplify tests with requests_mock_datadir fixture
    
    Use mocked network requests to get jar and pom files instead of
    reading them from the datadir directory.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/1022/ for more details.

This revision is now accepted and ready to land.Nov 14 2022, 5:43 PM