HomeSoftware Heritage

packagist: Reimplement lister using new Lister API

Description

packagist: Reimplement lister using new Lister API

The previous implementation was generating tasks for a non implemented
Packagist loader.

The new implementation extracts source repository URL, VCS type and
last update date for each package referenced by Packagist and send
those info to the scheduler.

Packages metadata are retrieved using Packagist API endpoints whose
responses are served from static files, which are guaranteed to be
efficient on the Packagist side (no dymamic queries).
Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
header to only retrieve packages metadata updated since the previous
listing operation in order to save bandwidth and return only origins
which might have new released versions.

Closes T2991

Details

Provenance
anlambertAuthored on Feb 1 2021, 5:34 PM
anlambertPushed on Feb 2 2021, 4:08 PM
Differential Revision
D4990: packagist: Reimplement lister using new Lister API
Parents
rDLS82ab96ad0671: gnu: Remove dependency on pytz
Branches
Unknown
Tags
Unknown
Tasks
T2991: Port packagist lister to the new Lister API
Build Status
Buildable 18960
Build 29379: test-and-buildJenkins console · Jenkins