Paths

Table of Contentst

Differential D4990

packagist: Reimplement lister using new Lister API
ClosedPublic
Actions

Authored by anlambert on Feb 2 2021, 11:04 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2991: Port packagist lister to the new Lister API

Commits

rDLSff05191b7db7: packagist: Reimplement lister using new Lister API

Summary

The previous implementation was generating tasks for a non implemented
Packagist loader.

The new implementation extracts source repository URL, VCS type and
last update date for each package referenced by Packagist and send
those info to the scheduler.

Packages metadata are retrieved using Packagist API endpoints whose
responses are served from static files, which are guaranteed to be
efficient on the Packagist side (no dymamic queries).
Furthermore, subsequent listing will send the If-Modified-Since HTTP
header to only retrieve packages metadata updated since the previous
listing operation in order to save bandwidth and return only origins
which might have new released versions.

I tested intensively the lister yersteday and it worked without any
issues each time I executed it. First execution took around 90 minutes
and listed 286510 origins with three different visit types: git, hg and
svn. Subsequent calls took less time thanks to the If-Mofified-Since
HTTP header use and only returned packages modified since last listing.

Closes T2991

Diff Detail

Repository

rDLS Listers

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Feb 2 2021, 11:04 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 2 2021, 11:04 AM

Build is green

Patch application report for D4990 (id=17798)

Rebasing onto 8e4dd178f1...

Current branch diff-target is up to date.

Changes applied before test

commit 478081c1513b240f85c78cc66e9a3109eff91608
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Feb 1 17:34:10 2021 +0100

    packagist: Reimplement lister using new Lister API
    
    The previous implementation was generating tasks for a non implemented
    Packagist loader.
    
    The new implementation extracts source repository URL, VCS type and
    last update date for each package referenced by Packagist and send
    those info to the scheduler.
    
    Packages metadata are retrieved using Packagist API endpoints whose
    responses are served from static files, which are guaranteed to be
    efficient on the Packagist side (no dymamic queries).
    Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
    header to only retrieve packages metadata updated since the previous
    listing operation in order to save bandwidth and return only origins
    which might have new released versions.
    
    Closes T2991

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/234/ for more details.

Harbormaster completed remote builds in B18939: Diff 17798.Feb 2 2021, 11:07 AM

anlambert requested review of this revision.Feb 2 2021, 11:07 AM

anlambert added a child revision: D4992: Remove no longer used legacy Lister API and update CLI options.Feb 2 2021, 12:45 PM

lgtm

But it's missing some coverage on conditionals (according to jenkins).

Maybe simply enrich the current test dataset with some of
those skipped packages in the current new dataset you added
(one bitbucket entry, another with missing origin_url, another
with missing time, etc...)

swh/lister/packagist/lister.py
3	lol (a regexp change gone rogue ;)
29	when*

In D4990#125787, @ardumont wrote:

lgtm

But it's missing some coverage on conditionals (according to jenkins).

Maybe simply enrich the current test dataset with some of
those skipped packages in the current new dataset you added
(one bitbucket entry, another with missing origin_url, another
with missing time, etc...)

Ack, will improve coverage then.

swh/lister/packagist/lister.py
3	lol, thanks for spotting !

Rebase and improve coverage

Build is green

Patch application report for D4990 (id=17810)

Rebasing onto 82ab96ad06...

Current branch diff-target is up to date.

Changes applied before test

commit ff05191b7db7b217c8682e9888338b8813e2df6a
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Mon Feb 1 17:34:10 2021 +0100

    packagist: Reimplement lister using new Lister API
    
    The previous implementation was generating tasks for a non implemented
    Packagist loader.
    
    The new implementation extracts source repository URL, VCS type and
    last update date for each package referenced by Packagist and send
    those info to the scheduler.
    
    Packages metadata are retrieved using Packagist API endpoints whose
    responses are served from static files, which are guaranteed to be
    efficient on the Packagist side (no dymamic queries).
    Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
    header to only retrieve packages metadata updated since the previous
    listing operation in order to save bandwidth and return only origins
    which might have new released versions.
    
    Closes T2991

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/238/ for more details.

Harbormaster completed remote builds in B18952: Diff 17810.Feb 2 2021, 2:52 PM

Thanks.

(I had forgotten to actually validate it ¯\_(ツ)_/¯ )

This revision is now accepted and ready to land.Feb 2 2021, 3:00 PM

Closed by commit rDLSff05191b7db7: packagist: Reimplement lister using new Lister API (authored by anlambert). · Explain WhyFeb 2 2021, 4:08 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLSff05191b7db7: packagist: Reimplement lister using new Lister API.