Note that the current implementation will start back the new visit from the last
next_page link seen (that's what is stored in the lister state to avoid computing back
the url). This means that this page will be seen at least 2 times, on the first visit
and on the next. This should not pose any problems as the listing is idempotent.
Details
Details
- Reviewers
anlambert - Group Reviewers
Reviewers - Maniphest Tasks
- T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API
- Commits
- rDLS1f911401a1bb: gitlab: Add test on incremental implementation
Diff Detail
Diff Detail
- Repository
- rDLS Listers
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
Comment Actions
Build is green
Patch application report for D4932 (id=17543)
Could not rebase; Attempt merge onto ff232f0d91...
Updating ff232f0..1f91140
Fast-forward
swh/lister/gitlab/__init__.py | 4 +-
swh/lister/gitlab/lister.py | 258 +++++++++++++++------
swh/lister/gitlab/models.py | 18 --
swh/lister/gitlab/tests/conftest.py | 26 ---
.../https_gite.lirmm.fr/api_response_page1.json | 42 ++++
.../https_gite.lirmm.fr/api_response_page2.json | 42 ++++
.../https_gite.lirmm.fr/api_response_page3.json | 22 ++
.../data/https_gitlab.com/api_empty_response.json | 1 -
.../tests/data/https_gitlab.com/api_response.json | 1 -
...ort=asc,per_page=20 => api_response_page1.json} | 0
swh/lister/gitlab/tests/test_lister.py | 200 ++++++++++++----
swh/lister/tests/test_cli.py | 1 -
12 files changed, 444 insertions(+), 171 deletions(-)
delete mode 100644 swh/lister/gitlab/models.py
delete mode 100644 swh/lister/gitlab/tests/conftest.py
create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page3.json
delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)Changes applied before test
commit 1f911401a1bb3d1f792e843689cf9ecad1d30475
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Sat Jan 23 17:15:01 2021 +0100
gitlab: Add test on incremental implementation
Note that the current implementation will start back the new visit from the last
next_page link seen (that's what is stored in the lister state to avoid computing back
the url). This means that this page will be seen at least 2 times, on the first visit
and on the next. This should not pose any problems as the listing is idempotent.
Related to T2987
commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Sat Jan 23 15:51:18 2021 +0100
gitlab: Add test on pagination
Related to T2987
commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Sat Jan 23 15:18:20 2021 +0100
gitlab: Port to the new lister api
Related to T2987See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/142/ for more details.