HomeSoftware Heritage

lister: Add utility decorator to ease HTTP requests rate limit handling

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

lister: Add utility decorator to ease HTTP requests rate limit handling

Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.

Details

Provenance
anlambertAuthored on Jan 15 2021, 12:25 PM
anlambertPushed on Jan 18 2021, 11:29 AM
Differential Revision
D4869: lister: Add utility module to ease HTTP requests rate limit handling
Build Status
Buildable 18437
Build 28504: test-and-buildJenkins console · Jenkins

Commit No Longer Exists

This commit no longer exists in the repository.