HomeSoftware Heritage

Refactor and deduplicate HTTP requests code in listers

Description

Refactor and deduplicate HTTP requests code in listers

Numerous listers were using the same page_request method or equivalent
in their implementation so prefer to deduplicate that code by adding
an http_request method in base lister class: swh.lister.pattern.Lister.

That method simply wraps a call to requests.Session.request and logs
some useful info for debugging and error reporting, also an HTTPError
will be raised if a request ends up with an error.

All listers using that new method now benefit of requests retry when
an HTTP error occurs thanks to the use of the http_retry decorator.

Details

Provenance
anlambertAuthored on Sep 21 2022, 7:53 PM
anlambertPushed on Sep 26 2022, 11:36 AM
Differential Revision
D8520: Refactor and deduplicate HTTP requests code in listers
Parents
rDLS9c55acd28609: Use generic HTTP retry policy by default and rename dedicated decorator
Branches
Unknown
Tags
Unknown
Build Status
Buildable 31722
Build 49635: test-and-buildJenkins console · Jenkins