Page MenuHomeSoftware Heritage

gitlab: Add test on rate-limit support
ClosedPublic

Authored by ardumont on Jan 25 2021, 9:23 AM.

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D4933 (id=17545)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..b352b8e
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 258 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page3.json    |  22 ++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 244 +++++++++++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 12 files changed, 488 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page3.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit b352b8e11ebe2840f45e24749510443121221d79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Jan 25 09:23:22 2021 +0100

    gitlab: Add test on rate-limit support
    
    Related to T2987

commit 1f911401a1bb3d1f792e843689cf9ecad1d30475
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 17:15:01 2021 +0100

    gitlab: Add test on incremental implementation
    
    Note that the current implementation will start back the new visit from the last
    next_page link seen (that's what is stored in the lister state to avoid computing back
    the url). This means that this page will be seen at least 2 times, on the first visit
    and on the next. This should not pose any problems as the listing is idempotent.
    
    Related to T2987

commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/144/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me !

swh/lister/gitlab/tests/test_lister.py
16

Nitpick: now that the function is imported in another test module, we should remove the _ prefix.

This revision is now accepted and ready to land.Jan 25 2021, 12:33 PM
This revision was automatically updated to reflect the committed changes.