Page MenuHomeSoftware Heritage

gitlab: Add test on incremental implementation
ClosedPublic

Authored by ardumont on Jan 25 2021, 8:53 AM.

Details

Summary

Note that the current implementation will start back the new visit from the last
next_page link seen (that's what is stored in the lister state to avoid computing back
the url). This means that this page will be seen at least 2 times, on the first visit
and on the next. This should not pose any problems as the listing is idempotent.

Related to T2987
Depends on D4930

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D4932 (id=17543)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..1f91140
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 258 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page3.json    |  22 ++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 200 ++++++++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 12 files changed, 444 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page3.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit 1f911401a1bb3d1f792e843689cf9ecad1d30475
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 17:15:01 2021 +0100

    gitlab: Add test on incremental implementation
    
    Note that the current implementation will start back the new visit from the last
    next_page link seen (that's what is stored in the lister state to avoid computing back
    the url). This means that this page will be seen at least 2 times, on the first visit
    and on the next. This should not pose any problems as the listing is idempotent.
    
    Related to T2987

commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/142/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me !

This revision is now accepted and ready to land.Jan 25 2021, 12:35 PM