Page MenuHomeSoftware Heritage

gitlab: Handle HTTP status code 500 when listing projects
ClosedPublic

Authored by anlambert on Jul 23 2021, 2:37 PM.

Details

Summary

GitLab API can return errors 500 when listing projects
(see https://gitlab.com/gitlab-org/gitlab/-/issues/262629).

To avoid ending the listing prematurely, skip buggy URLs and move
to next pages.

Related to T3442

Depends on D6021

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6022 (id=21762)

Could not rebase; Attempt merge onto 73f85c0b8a...

Updating 73f85c0..35ced44
Fast-forward
 swh/lister/gitlab/lister.py            | 23 ++++++++++++++++++++++-
 swh/lister/gitlab/tests/test_lister.py | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)
Changes applied before test
commit 35ced44b52a0b67f673fe6c6d6f8c698401d3e72
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 23 14:33:15 2021 +0200

    gitlab: Handle HTTP status code 500 when listing projects
    
    GitLab API can return errors 500 when listing projects
    (see https://gitlab.com/gitlab-org/gitlab/-/issues/262629).
    
    To avoid ending the listing prematurely, skip buggy URLs and move
    to next pages.
    
    Related to T3442

commit 52c315015543340c9647174f206aabcfa438feb4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 23 13:59:52 2021 +0200

    gitlab: Update requests query parameters
    
    Increase number of origins per page to the maximum value allowed
    by GitLab API (100) to send less requests.
    
    Ask for simple responses to reduce size of JSON data.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/335/ for more details.

ardumont added a subscriber: ardumont.

lgtm

one typo inline ;)

swh/lister/gitlab/tests/test_lister.py
247
This revision is now accepted and ready to land.Jul 23 2021, 2:59 PM

Build is green

Patch application report for D6022 (id=21763)

Could not rebase; Attempt merge onto 73f85c0b8a...

Updating 73f85c0..e904f47
Fast-forward
 swh/lister/gitlab/lister.py            | 23 ++++++++++++++++++++++-
 swh/lister/gitlab/tests/test_lister.py | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)
Changes applied before test
commit e904f4760ef10a99fc5c89bfb5cd4e28e6efbd95
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 23 14:33:15 2021 +0200

    gitlab: Handle HTTP status code 500 when listing projects
    
    GitLab API can return errors 500 when listing projects
    (see https://gitlab.com/gitlab-org/gitlab/-/issues/262629).
    
    To avoid ending the listing prematurely, skip buggy URLs and move
    to next pages.
    
    Related to T3442

commit 52c315015543340c9647174f206aabcfa438feb4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 23 13:59:52 2021 +0200

    gitlab: Update requests query parameters
    
    Increase number of origins per page to the maximum value allowed
    by GitLab API (100) to send less requests.
    
    Ask for simple responses to reduce size of JSON data.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/336/ for more details.