Page MenuHomeSoftware Heritage
Differential D8615

cpan: Improve listing process by querying the metacpan release endpoint
ClosedPublic
Actions

Authored by anlambert on Oct 4 2022, 5:17 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2833: cpan.loader - archive Perl modules from CPAN
Commits
rDLSf57b8f3a2c49: cpan: Improve listing process by querying the metacpan release endpoint
Summary

Instead of querying the metacpan distribution endpoint to list origins,
prefer to use the release endpoint instead enabling to list all artifacts
associated to CPAN packages by scrolling results.

Compared to previous implementation, it enables to compute a last_update
date for all CPAN packages but also to obtain artifact sha256 checksums
that will be used by the CPAN loader to check downloads integrity.

It also enables to save a call to metacapan Web API in cpan loader as all
needed info about package artifacts are now provided as extra loader
arguments.

Related to T2833

When testing this in docker, I could list all CPAN packages and their
artifacts in less than 4 minutes.

Diff Detail

Repository
rDLS Listers
Branch
cpan-lister-improvements
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32204
Build 50435: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50434: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.cpan.lister::swh.lister.cpan.lister.get_field_value
025 026 Splits ``field_name`` on ``.``, and use it as path in the nested ``entry`` 027 dictionary. If a value does not exist, returns None.
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.lister::swh.lister.arch.lister.size_to_bytes
2,161 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_lister::test_arch_lister
507 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_lister
3,762 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_ping
View Full Test Results (1 Failed · 329 Passed)

Event Timeline

anlambert created this revision.Oct 4 2022, 5:17 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 4 2022, 5:17 PM
swh-public-ci added a comment.Oct 4 2022, 5:22 PM

Build is green

Patch application report for D8615 (id=31113)

Rebasing onto 5daead68ad...

Current branch diff-target is up to date.
Changes applied before test
commit fdfd876de96b91d4adc27f81f7145b3d74508eca
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    It also enables to save a call to metacapan Web API in cpan loader as all
    needed info about package artifacts are now provided as extra loader
    arguments.
    
    Related to T2833

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/748/ for more details.

Harbormaster completed remote builds in B32083: Diff 31113.Oct 4 2022, 5:22 PM
anlambert requested review of this revision.Oct 4 2022, 5:22 PM
anlambert added a child revision: D8616: cpan: Align loader implementation with latest lister improvements.Oct 4 2022, 5:24 PM
anlambert planned changes to this revision.Oct 7 2022, 12:30 PM
anlambert updated this revision to Diff 31230.Oct 10 2022, 4:25 PM

Rebase and update cpan lister:

  • process all pages from CPAN API before sending origins to scheduler
  • add extrinsic metdata URL for each module version and send it to cpan loader
  • miscelleaenous code improvements
anlambert added a child revision: D8648: cpan: Fix module version extraction for some edge cases.Oct 10 2022, 4:26 PM
swh-public-ci added a comment.Oct 10 2022, 4:31 PM

Build is green

Patch application report for D8615 (id=31230)

Rebasing onto 108816f232...

Current branch diff-target is up to date.
Changes applied before test
commit 2177ac9f5a08c2bd276f494b2aa4c8f0d4239e65
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    As the multiple versions of a module are spread across multiple pages
    from the CPAN API, origins are sent to the scheduler once all pages
    processed, it is also faster to proceed that way.
    
    Also compute extrinsic metadata URL for each perl module versions in
    order for the cpan loader to query it.
    
    Related to T2833

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/774/ for more details.

Harbormaster completed remote builds in B32183: Diff 31230.Oct 10 2022, 4:31 PM
anlambert edited the summary of this revision. (Show Details)Oct 10 2022, 5:01 PM
vlorentz added a subscriber: vlorentz.Oct 11 2022, 9:42 AM
vlorentz added inline comments.
swh/lister/cpan/lister.py
83

and move it outside the class

86–87

easier to read, IMO

89
136–138

I'd rather pass the BASE_URL and let the loader build this URL; it will allow changing loader behavior without changing the lister too.

anlambert added inline comments.Oct 11 2022, 12:11 PM
swh/lister/cpan/lister.py
136–138

Ok but this means I have to add the release name in module metadata, not a big deal though.

anlambert updated this revision to Diff 31252.Oct 11 2022, 2:17 PM

Update: Address @vlorentz comments

swh-public-ci added a comment.Oct 11 2022, 2:22 PM

Build has FAILED

Patch application report for D8615 (id=31252)

Rebasing onto 108816f232...

Current branch diff-target is up to date.
Changes applied before test
commit 5042a43e31c091d186a7e38c36df0235f6cd65e7
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    As the multiple versions of a module are spread across multiple pages
    from the CPAN API, origins are sent to the scheduler once all pages
    processed, it is also faster to proceed that way.
    
    Related to T2833

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/777/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/777/console

Harbormaster failed remote builds in B32204: Diff 31252!Oct 11 2022, 2:22 PM
anlambert updated this revision to Diff 31258.Oct 11 2022, 3:02 PM

Fix docstring

swh-public-ci added a comment.Oct 11 2022, 3:07 PM

Build has FAILED

Patch application report for D8615 (id=31258)

Rebasing onto 108816f232...

Current branch diff-target is up to date.
Changes applied before test
commit 5121157ce326d32411e32f9f984f9a1f6e8710ae
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    As the multiple versions of a module are spread across multiple pages
    from the CPAN API, origins are sent to the scheduler once all pages
    processed, it is also faster to proceed that way.
    
    Related to T2833

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/779/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/779/console

Harbormaster failed remote builds in B32210: Diff 31258!Oct 11 2022, 3:07 PM
anlambert updated this revision to Diff 31260.Oct 11 2022, 3:14 PM

really fix docstring

vlorentz accepted this revision.Oct 11 2022, 3:16 PM
This revision is now accepted and ready to land.Oct 11 2022, 3:16 PM
swh-public-ci added a comment.Oct 11 2022, 3:20 PM

Build is green

Patch application report for D8615 (id=31260)

Rebasing onto 108816f232...

Current branch diff-target is up to date.
Changes applied before test
commit e09a31c4c0072ff93453215aa772a7cfcabec5f1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    As the multiple versions of a module are spread across multiple pages
    from the CPAN API, origins are sent to the scheduler once all pages
    processed, it is also faster to proceed that way.
    
    Related to T2833

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/781/ for more details.

Harbormaster completed remote builds in B32212: Diff 31260.Oct 11 2022, 3:20 PM
anlambert updated this revision to Diff 31262.Oct 11 2022, 3:25 PM

remove no longer needed variable

swh-public-ci added a comment.Oct 11 2022, 3:30 PM

Build is green

Patch application report for D8615 (id=31262)

Rebasing onto 108816f232...

Current branch diff-target is up to date.
Changes applied before test
commit f57b8f3a2c49080ae9bc11217b8d6ef4ed8c564e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Sep 27 16:34:38 2022 +0200

    cpan: Improve listing process by querying the metacpan release endpoint
    
    Instead of querying the metacpan distribution endpoint to list origins,
    prefer to use the release endpoint instead enabling to list all artifacts
    associated to CPAN packages by scrolling results.
    
    Compared to previous implementation, it enables to compute a last_update
    date for all CPAN packages but also to obtain artifact sha256 checksums
    that will be used by the CPAN loader to check downloads integrity.
    
    As the multiple versions of a module are spread across multiple pages
    from the CPAN API, origins are sent to the scheduler once all pages
    processed, it is also faster to proceed that way.
    
    Related to T2833

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/783/ for more details.

Harbormaster completed remote builds in B32214: Diff 31262.Oct 11 2022, 3:30 PM
Closed by commit rDLSf57b8f3a2c49: cpan: Improve listing process by querying the metacpan release endpoint (authored by anlambert). · Explain WhyOct 11 2022, 5:15 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLSf57b8f3a2c49: cpan: Improve listing process by querying the metacpan release endpoint.
anlambert mentioned this in D8616: cpan: Align loader implementation with latest lister improvements.Oct 13 2022, 10:47 AM

Revision Contents
Changeset List

Diff 31252

View Options

swh/lister/cpan/__init__.py

Loading...
View Options

swh/lister/cpan/lister.py

Loading...
View Options

swh/lister/cpan/tests/data/https_fastapi.metacpan.org/v1__search_scroll

Loading...
View Options

swh/lister/cpan/tests/data/https_fastapi.metacpan.org/v1__search_scroll,scroll=1m,scroll_id=cXVlcnlUaGVuRmV0Y2g7Mzs5NTU1MTQ1NTk6eXptdmszQUNUam1XbVJjRjRkRk9Udzs5NTQ5NjQ5NjI6ZHZIZWxCb3BUZi1Cb3NwRDB5NmRQUTs5NTU1MTQ1NjA6eXptdmszQUNUam1XbVJjRjRkRk9UdzswOw==

Loading...
View Options

swh/lister/cpan/tests/data/https_fastapi.metacpan.org/v1__search_scroll,scroll=1m,scroll_id=cXVlcnlUaGVuRmV0Y2g7Mzs5NTU1MTQ1NTk6eXptdmszQUNUam1XbVJjRjRkRk9Udzs5NTQ5NjQ5NjI6ZHZIZWxCb3BUZi1Cb3NwRDB5NmRQUTs5NTU1MTQ1NjA6eXptdmszQUNUam1XbVJjRjRkRk9UdzswOw==_visit1

Loading...
View Options

swh/lister/cpan/tests/data/https_fastapi.metacpan.org/v1_distribution__search,fields=name,size=1000,scroll=1m

Loading...
View Options

swh/lister/cpan/tests/data/https_fastapi.metacpan.org/v1_release__search

Loading...
View Options

swh/lister/cpan/tests/test_lister.py

Loading...
About · Developer information · Legal