Page MenuHomeSoftware Heritage
Differential D4869

lister: Add utility module to ease HTTP requests rate limit handling
ClosedPublic
Actions

Authored by anlambert on Jan 15 2021, 12:42 PM.

Details

Reviewers
tenma
Group Reviewers
Reviewers
Commits
rDLSd1fbccd988e5: lister: Add utility decorator to ease HTTP requests rate limit handling
Summary

Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.

I also ensured that introduced code can be executed and tested with the
tenacity version packaged in debian buster (4.12).

Example of use:

import requests

from swh.lister.utils import throttling_retry

URL = "http://example.org/api/repositories"

@throttling_retry()
def make_request():
    response = requests.get(URL)
    response.raise_for_status()
    return response

Diff Detail

Repository
rDLS Listers
Branch
lister-request-throttling
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18435
Build 28501: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28500: arc lint + arc unit

Event Timeline

anlambert created this revision.Jan 15 2021, 12:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 15 2021, 12:42 PM
Harbormaster completed remote builds in B18394: Diff 17252.Jan 15 2021, 12:45 PM
swh-public-ci added a comment.Jan 15 2021, 12:45 PM

Build is green

Patch application report for D4869 (id=17252)

Rebasing onto c782275296...

Current branch diff-target is up to date.
Changes applied before test
commit f81482a62bbcbe2dddc029b18c2ff9c5d69623ac
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 12:25:45 2021 +0100

    lister: Add utility module to ease HTTP requests rate limit handling
    
    Add throttling_retry module providing a decorator enabling to retry a
    function that performs an HTTP request who can return a 429 status code.
    
    The implementation is based on the tenacity module and it is assumed
    that the requests library is used when querying an URL.
    
    The default wait strategy is based on exponential backoff.
    
    The default max number of attempts is set to 5, HTTPError exception
    will then be reraised.
    
    All tenacity.retry parameters can also be overridden in client code.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/101/ for more details.

anlambert requested review of this revision.Jan 15 2021, 12:45 PM
anlambert updated this revision to Diff 17265.Jan 15 2021, 2:07 PM

Update: Use requests.status_code.codes in tests

swh-public-ci added a comment.Jan 15 2021, 2:10 PM

Build is green

Patch application report for D4869 (id=17265)

Rebasing onto c782275296...

Current branch diff-target is up to date.
Changes applied before test
commit 7b70dbd4cd9620cf566f03fd10b7a8805a5df85f
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 12:25:45 2021 +0100

    lister: Add utility module to ease HTTP requests rate limit handling
    
    Add throttling_retry module providing a decorator enabling to retry a
    function that performs an HTTP request who can return a 429 status code.
    
    The implementation is based on the tenacity module and it is assumed
    that the requests library is used when querying an URL.
    
    The default wait strategy is based on exponential backoff.
    
    The default max number of attempts is set to 5, HTTPError exception
    will then be reraised.
    
    All tenacity.retry parameters can also be overridden in client code.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/103/ for more details.

Harbormaster completed remote builds in B18405: Diff 17265.Jan 15 2021, 2:10 PM
tenma accepted this revision.Jan 15 2021, 2:42 PM
tenma added a subscriber: tenma.

It would be great to parametrize the request and error type, but unfortunately it does not seem easy to do...

TGTM otherwise.

This revision is now accepted and ready to land.Jan 15 2021, 2:42 PM
anlambert added a comment.EditedJan 15 2021, 3:06 PM
In D4869#121891, @tenma wrote:

It would be great to parametrize the request and error type, but unfortunately it does not seem easy to do...

TGTM otherwise.

Actually, we can. For instance, the retry predicate for the github lister could be written and used like this:

from swh.lister.utils import is_throttling_exception, retry_attempt, throttling_retry

def github_is_anomymous_throttling(e: Exception) -> bool:
    return (
        isinstance(e, HTTPError) and e.response.status_code == 403
        and "Authorization" not in e.response.request.headers
    )

def github_retry_if_throttling(retry_state) -> bool:
    attempt = retry_attempt(retry_state)
    if attempt.failed:
        exception = attempt.exception()
        return is_throttling_exception(exception) or github_is_anomymous_throttling(exception)
    return False

@throttling_retry(retry=github_retry_if_throttling)
def make_github_request():
    ...
anlambert updated this revision to Diff 17296.Jan 18 2021, 10:54 AM

Update:

  • move code to swh.lister.utils module
  • Improve doctring
  • Add tenacity in requirements.txt
anlambert edited the summary of this revision. (Show Details)Jan 18 2021, 10:57 AM
swh-public-ci added a comment.Jan 18 2021, 10:58 AM

Build is green

Patch application report for D4869 (id=17296)

Rebasing onto c782275296...

Current branch diff-target is up to date.
Changes applied before test
commit 067553e30cb2bc705de0f07d048fb9c1ca7b90be
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 12:25:45 2021 +0100

    lister: Add utility module to ease HTTP requests rate limit handling
    
    Add throttling_retry module providing a decorator enabling to retry a
    function that performs an HTTP request who can return a 429 status code.
    
    The implementation is based on the tenacity module and it is assumed
    that the requests library is used when querying an URL.
    
    The default wait strategy is based on exponential backoff.
    
    The default max number of attempts is set to 5, HTTPError exception
    will then be reraised.
    
    All tenacity.retry parameters can also be overridden in client code.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/110/ for more details.

Harbormaster completed remote builds in B18435: Diff 17296.Jan 18 2021, 10:58 AM
anlambert updated this revision to Diff 17297.Jan 18 2021, 11:15 AM

Update commit message

anlambert edited the summary of this revision. (Show Details)Jan 18 2021, 11:16 AM
swh-public-ci added a comment.Jan 18 2021, 11:18 AM

Build is green

Patch application report for D4869 (id=17297)

Rebasing onto c782275296...

Current branch diff-target is up to date.
Changes applied before test
commit d2a02bc07822b39742464626fe15456967b3165e
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 12:25:45 2021 +0100

    lister: Add utility decorator to ease HTTP requests rate limit handling
    
    Add swh.lister.utils.throttling_retry decorator enabling to retry a
    function that performs an HTTP request who can return a 429 status code.
    
    The implementation is based on the tenacity module and it is assumed
    that the requests library is used when querying an URL.
    
    The default wait strategy is based on exponential backoff.
    
    The default max number of attempts is set to 5, HTTPError exception
    will then be reraised.
    
    All tenacity.retry parameters can also be overridden in client code.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/111/ for more details.

Harbormaster completed remote builds in B18436: Diff 17297.Jan 18 2021, 11:19 AM
Closed by commit rDLSd1fbccd988e5: lister: Add utility decorator to ease HTTP requests rate limit handling (authored by anlambert). · Explain WhyJan 18 2021, 11:29 AM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLSd1fbccd988e5: lister: Add utility decorator to ease HTTP requests rate limit handling.
tenma mentioned this in D4843: Reimplement Bitbucket lister using new Lister API.Jan 18 2021, 7:43 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
3 lines--
swh/
lister/
tests/
98 lines96%
Loading...
78 lines93%
Loading...

Diff 17296

View Options

requirements.txt

Loading...
View Options

swh/lister/tests/test_utils.py

Loading...
View Options

swh/lister/utils.py

Loading...
About · Developer information · Legal