Page MenuHomeSoftware Heritage

cpan: Align loader implementation with latest lister improvements
ClosedPublic

Authored by anlambert on Oct 4 2022, 5:24 PM.

Details

Summary

Artifacts info for a package are now provided as loader arguments so
no need to query metacpan Web API anymore.

Also remove package description in each release message as it does not
describe a release but rather the package.

Related to T2833

Depends on D8615
Depends on D8650

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
cpan-loader-improvements
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32187
Build 50405: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50404: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8616 (id=31114)

Rebasing onto b26b98810d...

Current branch diff-target is up to date.
Changes applied before test
commit 213add6830e132cb80bee218b09809bec29551ef
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Sep 29 20:36:38 2022 +0200

    cpan: Align loader implementation with latest lister improvements
    
    Artifacts info for a package are now provided as loader arguments so
    no need to query metacpan Web API anymore.
    
    Also remove package description in each release message as it does not
    describe a release but rather the package.
    
    Related to T2833

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/955/ for more details.

Also remove package description in each release message as it does not
describe a release but rather the package.

Please mention it in the diff/commit title, or make it its own commit. It's a meaningful change that should be visible.

swh/loader/package/cpan/loader.py
44–45

Rebase and move description removal from release message in separate commit and diff

Build is green

Patch application report for D8616 (id=31234)

Could not rebase; Attempt merge onto 028b7c04b9...

Updating 028b7c0..819f9d2
Fast-forward
 swh/loader/package/cpan/loader.py                  |  87 +++++++---------
 swh/loader/package/cpan/tests/data/fake_cpan.sh    |  86 ----------------
 .../v1_release_versions_Internals-CountObjects     |  26 -----
 swh/loader/package/cpan/tests/test_cpan.py         | 111 +++++++++++++++------
 swh/loader/package/cpan/tests/test_tasks.py        |   8 +-
 5 files changed, 119 insertions(+), 199 deletions(-)
 delete mode 100644 swh/loader/package/cpan/tests/data/fake_cpan.sh
 delete mode 100644 swh/loader/package/cpan/tests/data/https_fastapi.metacpan.org/v1_release_versions_Internals-CountObjects
Changes applied before test
commit 819f9d2702c193497a9ed99b17d58192aeb4ab9b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Sep 29 20:36:38 2022 +0200

    cpan: Align loader implementation with latest lister improvements
    
    Artifacts info for a package are now provided as loader arguments so
    no need to query metacpan Web API anymore to get list of versions
    and their related info.
    
    Related to T2833

commit e53a1e17aad238e38690a8a42b3672d106e179ae
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Oct 7 15:33:35 2022 +0200

    cpan: Remove module description from release message
    
    Module description is not related to a particular release so we
    should not add it in release message.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/982/ for more details.

This revision is now accepted and ready to land.Oct 11 2022, 10:31 AM

Rebase and synchronize again with latest cpan lister changes

Build is green

Patch application report for D8616 (id=31255)

Could not rebase; Attempt merge onto 4cb85e153e...

Updating 4cb85e1..a13e3e6
Fast-forward
 swh/loader/package/cpan/loader.py                  |  89 +++++++---------
 swh/loader/package/cpan/tests/data/fake_cpan.sh    |  86 ----------------
 .../v1_release_versions_Internals-CountObjects     |  26 -----
 swh/loader/package/cpan/tests/test_cpan.py         | 113 +++++++++++++++------
 swh/loader/package/cpan/tests/test_tasks.py        |  14 ++-
 5 files changed, 129 insertions(+), 199 deletions(-)
 delete mode 100644 swh/loader/package/cpan/tests/data/fake_cpan.sh
 delete mode 100644 swh/loader/package/cpan/tests/data/https_fastapi.metacpan.org/v1_release_versions_Internals-CountObjects
Changes applied before test
commit a13e3e6f35bcabf856664ad7f116b17ca5a3daaf
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Sep 29 20:36:38 2022 +0200

    cpan: Align loader implementation with latest lister improvements
    
    Artifacts info for a package are now provided as loader arguments so
    no need to query metacpan Web API anymore to get list of versions
    and their related info.
    
    Related to T2833

commit e17ee9e08e84105710852bcc32c81bf149e55d4c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Oct 7 15:33:35 2022 +0200

    cpan: Remove module description from release message
    
    Module description is not related to a particular release so we
    should not add it in release message.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/990/ for more details.

franckbret added inline comments.
swh/loader/package/cpan/loader.py
72

api_base_url should be removed too?

swh/loader/package/cpan/loader.py
72

indeed

swh/loader/package/cpan/loader.py
72

No, it is used in D8652 to compute extrinsic metadata URLs (@vlorentz, you asked me to do that computation in the loader instead of the lister, see D8615).

swh/loader/package/cpan/loader.py
72

my bad, I didn't connect the dots