Page MenuHomeSoftware Heritage
Differential D8523

Update value of User-Agent HTTP request header used by listers
ClosedPublic
Actions

Authored by anlambert on Sep 22 2022, 3:49 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDLSd5c30a3ce371: Update value of User-Agent HTTP request header used by listers
Summary

That HTTP header value will now contain the lister name but also a link
to our contact form in order for sysadmins to easily reach us if needed.

The following template is used to generate it:

"Software Heritage <lister_name> lister v<swh-lister version>
(+https://www.softwareheritage.org/contact)"

Depnds on D8520

Diff Detail

Repository
rDLS Listers
Branch
user-agent-improvement
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 31671
Build 49547: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 49546: arc lint + arc unit

Event Timeline

anlambert created this revision.Sep 22 2022, 3:49 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 22 2022, 3:49 PM
swh-public-ci added a comment.Sep 22 2022, 3:54 PM

Build is green

Patch application report for D8523 (id=30693)

Could not rebase; Attempt merge onto bd35d54398...

Updating bd35d54..c2a2f7b
Fast-forward
 docs/new_lister_template.py                 | 41 ++-------------------
 docs/tutorial.rst                           |  2 +-
 swh/lister/__init__.py                      | 14 ++++----
 swh/lister/arch/lister.py                   | 31 ++--------------
 swh/lister/aur/lister.py                    |  4 +--
 swh/lister/bitbucket/lister.py              | 33 +++--------------
 swh/lister/bitbucket/tests/test_lister.py   | 10 +++---
 swh/lister/bower/lister.py                  | 35 +++---------------
 swh/lister/cgit/lister.py                   | 15 ++------
 swh/lister/cgit/tests/test_lister.py        |  6 ++--
 swh/lister/debian/lister.py                 | 17 ++++-----
 swh/lister/gitea/tests/test_lister.py       |  2 +-
 swh/lister/github/lister.py                 |  4 +--
 swh/lister/gitlab/lister.py                 | 13 +++----
 swh/lister/gitlab/tests/test_lister.py      | 11 +++---
 swh/lister/gogs/lister.py                   | 52 ++++++++++-----------------
 swh/lister/gogs/tests/test_lister.py        |  8 ++---
 swh/lister/golang/lister.py                 | 28 ++-------------
 swh/lister/golang/tests/test_lister.py      |  9 ++---
 swh/lister/launchpad/lister.py              |  4 +--
 swh/lister/maven/lister.py                  | 56 +++++++++--------------------
 swh/lister/maven/tests/test_lister.py       |  5 +++
 swh/lister/npm/lister.py                    | 30 +++-------------
 swh/lister/npm/tests/test_lister.py         | 31 +++++++++++-----
 swh/lister/packagist/lister.py              | 25 +++----------
 swh/lister/pattern.py                       | 34 +++++++++++++++++-
 swh/lister/phabricator/lister.py            | 28 +++------------
 swh/lister/phabricator/tests/test_lister.py | 12 +++++--
 swh/lister/pubdev/lister.py                 | 43 ++++------------------
 swh/lister/pubdev/tests/test_lister.py      |  8 +++--
 swh/lister/pypi/lister.py                   |  6 ++--
 swh/lister/sourceforge/lister.py            | 43 +++++-----------------
 swh/lister/sourceforge/tests/test_lister.py | 16 +++++----
 swh/lister/tests/test_utils.py              | 32 +++++++++--------
 swh/lister/tuleap/lister.py                 | 39 ++++----------------
 swh/lister/tuleap/tests/test_lister.py      |  7 +++-
 swh/lister/utils.py                         | 14 ++------
 37 files changed, 252 insertions(+), 516 deletions(-)
Changes applied before test
commit c2a2f7ba588c5db2284f4ff5d41463824af291dd
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Sep 22 15:43:20 2022 +0200

    Update value of User-Agent HTTP request header used by listers
    
    That HTTP header value will now contain the lister name but also a link
    to our contact form in order for sysadmins to easily reach us if needed.
    
    The following template is used to generate it:
    
    "Software Heritage <lister_name> lister v<swh-lister version>
     (+https://www.softwareheritage.org/contact)"

commit d677988e22f0d71d6b4aa2dae24f2d41f2ce337a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 21 19:53:22 2022 +0200

    Refactor and deduplicate HTTP requests code in listers
    
    Numerous listers were using the same page_request method or equivalent
    in their implementation so prefer to deduplicate that code by adding
    an http_request method in base lister class: swh.lister.pattern.Lister.
    
    That method simply wraps a call to requests.Session.request and logs
    some useful info for debugging and error reporting, also an HTTPError
    will be raised if a request ends up with an error.
    
    All listers using that new method now benefit of requests retry when
    an HTTP error occurs thanks to the use of the http_retry decorator.

commit 6284d34b1725ce9c5e9f76359a2760da871edb7b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 21 16:56:34 2022 +0200

    Use generic HTTP retry policy by default and rename dedicated decorator
    
    Instead of retrying HTTP requests only for 429 status code by default,
    prefer to use the generic retry policy enabling to also retry for status
    codes >= 500 but also on ConnectionError exceptions.
    
    Rename throttling_retry decorator to http_retry to reflect this change.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/673/ for more details.

Harbormaster completed remote builds in B31671: Diff 30693.Sep 22 2022, 3:54 PM
anlambert requested review of this revision.Sep 22 2022, 3:54 PM
ardumont accepted this revision.Sep 23 2022, 9:40 AM
This revision is now accepted and ready to land.Sep 23 2022, 9:40 AM
anlambert updated this revision to Diff 30741.Sep 26 2022, 10:51 AM

Rebase

swh-public-ci added a comment.Sep 26 2022, 10:56 AM

Build is green

Patch application report for D8523 (id=30741)

Could not rebase; Attempt merge onto 9b3e565cf7...

Updating 9b3e565..d5c30a3
Fast-forward
 docs/new_lister_template.py                 | 41 ++-------------------
 docs/tutorial.rst                           |  2 +-
 swh/lister/__init__.py                      | 14 ++++----
 swh/lister/arch/lister.py                   | 31 ++--------------
 swh/lister/aur/lister.py                    |  4 +--
 swh/lister/bitbucket/lister.py              | 33 +++--------------
 swh/lister/bitbucket/tests/test_lister.py   | 10 +++---
 swh/lister/bower/lister.py                  | 35 +++---------------
 swh/lister/cgit/lister.py                   | 15 ++------
 swh/lister/cgit/tests/test_lister.py        |  6 ++--
 swh/lister/debian/lister.py                 | 17 ++++-----
 swh/lister/gitea/tests/test_lister.py       |  2 +-
 swh/lister/github/lister.py                 |  4 +--
 swh/lister/gitlab/lister.py                 | 13 +++----
 swh/lister/gitlab/tests/test_lister.py      | 11 +++---
 swh/lister/gogs/lister.py                   | 52 ++++++++++-----------------
 swh/lister/gogs/tests/test_lister.py        |  8 ++---
 swh/lister/golang/lister.py                 | 28 ++-------------
 swh/lister/golang/tests/test_lister.py      |  9 ++---
 swh/lister/launchpad/lister.py              |  4 +--
 swh/lister/maven/lister.py                  | 56 +++++++++--------------------
 swh/lister/maven/tests/test_lister.py       |  5 +++
 swh/lister/npm/lister.py                    | 30 +++-------------
 swh/lister/npm/tests/test_lister.py         | 31 +++++++++++-----
 swh/lister/packagist/lister.py              | 25 +++----------
 swh/lister/pattern.py                       | 34 +++++++++++++++++-
 swh/lister/phabricator/lister.py            | 28 +++------------
 swh/lister/phabricator/tests/test_lister.py | 12 +++++--
 swh/lister/pubdev/lister.py                 | 43 ++++------------------
 swh/lister/pubdev/tests/test_lister.py      |  8 +++--
 swh/lister/pypi/lister.py                   |  6 ++--
 swh/lister/sourceforge/lister.py            | 43 +++++-----------------
 swh/lister/sourceforge/tests/test_lister.py | 16 +++++----
 swh/lister/tests/test_utils.py              | 32 +++++++++--------
 swh/lister/tuleap/lister.py                 | 39 ++++----------------
 swh/lister/tuleap/tests/test_lister.py      |  7 +++-
 swh/lister/utils.py                         | 14 ++------
 37 files changed, 252 insertions(+), 516 deletions(-)
Changes applied before test
commit d5c30a3ce371efa73cbabece72ce214bf18e3d3e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Sep 22 15:43:20 2022 +0200

    Update value of User-Agent HTTP request header used by listers
    
    That HTTP header value will now contain the lister name but also a link
    to our contact form in order for sysadmins to easily reach us if needed.
    
    The following template is used to generate it:
    
    "Software Heritage <lister_name> lister v<swh-lister version>
     (+https://www.softwareheritage.org/contact)"

commit db6ce12e9e3bf527e81a3c8fb3b159d81f6a7d0f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 21 19:53:22 2022 +0200

    Refactor and deduplicate HTTP requests code in listers
    
    Numerous listers were using the same page_request method or equivalent
    in their implementation so prefer to deduplicate that code by adding
    an http_request method in base lister class: swh.lister.pattern.Lister.
    
    That method simply wraps a call to requests.Session.request and logs
    some useful info for debugging and error reporting, also an HTTPError
    will be raised if a request ends up with an error.
    
    All listers using that new method now benefit of requests retry when
    an HTTP error occurs thanks to the use of the http_retry decorator.

commit 9c55acd286091acb6f6094e9fe1c95aca1fdeeec
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Sep 21 16:56:34 2022 +0200

    Use generic HTTP retry policy by default and rename dedicated decorator
    
    Instead of retrying HTTP requests only for 429 status code by default,
    prefer to use the generic retry policy enabling to also retry for status
    codes >= 500 but also on ConnectionError exceptions.
    
    Rename throttling_retry decorator to http_retry to reflect this change.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/686/ for more details.

Harbormaster completed remote builds in B31718: Diff 30741.Sep 26 2022, 10:56 AM
Closed by commit rDLSd5c30a3ce371: Update value of User-Agent HTTP request header used by listers (authored by anlambert). · Explain WhySep 26 2022, 11:36 AM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLSd5c30a3ce371: Update value of User-Agent HTTP request header used by listers.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
lister/
7 lines80%
Loading...
cgit/
tests/
2 lines100%
Loading...
github/
4 lines91%
Loading...
gitlab/
tests/
11 lines100%
Loading...
maven/
4 lines96%
Loading...
npm/
tests/
6 lines100%
Loading...
6 lines93%
Loading...
phabricator/
tests/
5 lines100%
Loading...
pubdev/
14 lines100%
Loading...
tests/
8 lines100%
Loading...
sourceforge/
tests/
7 lines99%
Loading...

Diff 30693

View Options

swh/lister/__init__.py

Loading...
View Options

swh/lister/cgit/tests/test_lister.py

Loading...
View Options

swh/lister/github/lister.py

Loading...
View Options

swh/lister/gitlab/tests/test_lister.py

Loading...
View Options

swh/lister/maven/lister.py

Loading...
View Options

swh/lister/npm/tests/test_lister.py

Loading...
View Options

swh/lister/pattern.py

Loading...
View Options

swh/lister/phabricator/tests/test_lister.py

Loading...
View Options

swh/lister/pubdev/lister.py

Loading...
View Options

swh/lister/pubdev/tests/test_lister.py

Loading...
View Options

swh/lister/sourceforge/tests/test_lister.py

Loading...
About · Developer information · Legal