Differential D8840

maven: Add support for md5 checkums to check download integrity
ClosedPublic
Actions

Authored by anlambert on Nov 14 2022, 4:26 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDLDBASE301502cbf8dc: maven: Add support for md5 checkums to check download integrity

Summary

Some maven artifacts do not have any sha1 sums computed but rather md5
ones so handle these edge cases to still check download integrity of
jar files.

See https://repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-test-annotations-common/1.5.30-M1/ for instance.

Depends on D8839

Fixes SWH-LOADER-CORE-1BP.

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Branch

maven-add-md5sum-support

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 32790
Build 51374: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 51373: arc lint + arc unit

Event Timeline

anlambert created this revision.Nov 14 2022, 4:26 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 14 2022, 4:27 PM

Build is green

Patch application report for D8840 (id=31854)

Could not rebase; Attempt merge onto bf2cb039d5...

Updating bf2cb03..301502c
Fast-forward
 swh/loader/package/maven/loader.py                 |  11 ++-
 .../sprova4j-0.1.0-sources.jar.sha1                |   1 -
 ...aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar} | Bin
 ...i_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar.md5 |   1 +
 ...ven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0.pom} |   0
 ...aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar} | Bin
 ...sprova4j_0.1.1_sprova4j-0.1.1-sources.jar.sha1} |   0
 ...ven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1.pom} |   0
 swh/loader/package/maven/tests/test_maven.py       | 107 +++++++--------------
 9 files changed, 43 insertions(+), 77 deletions(-)
 delete mode 100644 swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.0-sources.jar.sha1
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.0-sources.jar => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar} (100%)
 create mode 100644 swh/loader/package/maven/tests/data/https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0-sources.jar.md5
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.0.pom => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.0_sprova4j-0.1.0.pom} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1-sources.jar => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1-sources.jar.sha1 => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1-sources.jar.sha1} (100%)
 rename swh/loader/package/maven/tests/data/{https_maven.org/sprova4j-0.1.1.pom => https_repo1.maven.org/maven2_al_aldi_sprova4j_0.1.1_sprova4j-0.1.1.pom} (100%)

Changes applied before test

commit 301502cbf8dc5d69c01e86278161628b157c9c47
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 14 16:19:27 2022 +0100

    maven: Add support for md5 checkums to check download integrity
    
    Some maven artifacts do not have any sha1 sums computed but rather md5
    ones so handle these edge cases to still check download integrity of
    jar files.

commit 5778cfce008af543c566197f0c229637bceac287
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Nov 14 16:12:12 2022 +0100

    maven: Simplify tests with requests_mock_datadir fixture
    
    Use mocked network requests to get jar and pom files instead of
    reading them from the datadir directory.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/1022/ for more details.

Harbormaster completed remote builds in B32790: Diff 31854.Nov 14 2022, 4:30 PM

anlambert requested review of this revision.Nov 14 2022, 4:30 PM

anlambert edited the summary of this revision. (Show Details)Nov 14 2022, 4:33 PM

ardumont accepted this revision.Nov 14 2022, 5:43 PM

This revision is now accepted and ready to land.Nov 14 2022, 5:43 PM

Closed by commit rDLDBASE301502cbf8dc: maven: Add support for md5 checkums to check download integrity (authored by anlambert). · Explain WhyNov 15 2022, 11:48 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDBASE301502cbf8dc: maven: Add support for md5 checkums to check download integrity.