Page MenuHomeSoftware Heritage
Differential D4930

gitlab: Add tests on pagination
ClosedPublic
Actions

Authored by ardumont on Jan 23 2021, 4:00 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API
Commits
rDLS84dd616ab641: gitlab: Add test on pagination
Summary

This add one test with a test dataset to ensure pagination works as expected.

Related to T2987
Depends on D4929

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18673
Build 28899: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28898: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
684 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.debian.tests.test_lister::test_lister_debian
lister_debian = <swh.lister.debian.lister.DebianLister object at 0x7f3ce9b76e10> datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/lister/debian/tests/data' requests_mock_datadir = <requests_mock.mocker.Mocker object at 0x7f3ce9b83e48>
182 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_full_lister
398 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_incremental_lister
174 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_lister_rate_limit_hit
1,159 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_tasks::test_full_listing
View Full Test Results (1 Failed · 131 Passed)

Event Timeline

ardumont created this revision.Jan 23 2021, 4:00 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 23 2021, 4:00 PM
swh-public-ci added a comment.Jan 23 2021, 4:06 PM

Build is green

Patch application report for D4930 (id=17540)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..e11853e
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ojects,page=1,order_by=id,sort=asc,per_page=20} |   0
 swh/lister/gitlab/tests/test_lister.py             | 131 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 348 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_v4_projects,page=1,order_by=id,sort=asc,per_page=20} (100%)
Changes applied before test
commit e11853eb6e69cc7f38a9fec4d1dd11768911eca6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add tests on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/138/ for more details.

Harbormaster completed remote builds in B18672: Diff 17540.Jan 23 2021, 4:06 PM
ardumont requested review of this revision.Jan 23 2021, 4:06 PM
ardumont updated this revision to Diff 17541.Jan 23 2021, 4:13 PM

Refactor existing test to reuse the same mechanism as the new one

swh-public-ci added a comment.Jan 23 2021, 4:16 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/console

Harbormaster failed remote builds in B18673: Diff 17541!Jan 23 2021, 4:16 PM
swh-public-ci added a comment.Jan 23 2021, 4:29 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/console

swh-public-ci added a comment.Jan 23 2021, 4:39 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/console

ardumont edited the test plan for this revision. (Show Details)Jan 25 2021, 8:46 AM
ardumont added a task: T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API.
ardumont edited the summary of this revision. (Show Details)Jan 25 2021, 8:52 AM
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont added a parent revision: D4929: gitlab: Port to the new lister api.
ardumont added a child revision: D4932: gitlab: Add test on incremental implementation.Jan 25 2021, 8:53 AM
ardumont mentioned this in D4929: gitlab: Port to the new lister api.Jan 25 2021, 8:55 AM
ardumont updated this revision to Diff 17544.Jan 25 2021, 8:56 AM

Rebase

ardumont edited the summary of this revision. (Show Details)Jan 25 2021, 8:56 AM
swh-public-ci added a comment.Jan 25 2021, 8:59 AM

Build is green

Patch application report for D4930 (id=17544)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..84dd616
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B18678: Diff 17544.Jan 25 2021, 8:59 AM
anlambert accepted this revision.Jan 25 2021, 12:28 PM
anlambert added a subscriber: anlambert.

Looks good to me !

swh/lister/gitlab/tests/test_lister.py
24–25

Maybe you could add a page_url(page_id: int) in the lister to avoid duplicating URL construction in tests.
You could then write:

def test_lister_gitlab(datadir, swh_scheduler, requests_mock):
    """Gitlab lister supports full listing

    """
    instance = "gitlab.com"
    url = api_url(instance)

    response = gitlab_page_response(datadir, instance, 1)

    lister_gitlab = GitLabLister(
        swh_scheduler, url=url, instance=instance

    requests_mock.get(
        lister.page_url(1), [{"json": response}], additional_matcher=_match_request,
    )
This revision is now accepted and ready to land.Jan 25 2021, 12:28 PM
Closed by commit rDLS84dd616ab641: gitlab: Add test on pagination (authored by ardumont). · Explain WhyJan 25 2021, 12:54 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS84dd616ab641: gitlab: Add test on pagination.
ardumont added inline comments.Jan 25 2021, 1:22 PM
swh/lister/gitlab/tests/test_lister.py
24–25

good idea, yes.

Revision Contents
Changeset List

PathSize
swh/
lister/
gitlab/
tests/
data/
https_gite.lirmm.fr/
42 lines
42 lines
https_gitlab.com/
api_response_page1.json
api_v4_projects,page=1,order_by=id,sort=asc,per_page=20
170 lines
103 lines

Diff 17541

View Options

swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gitlab.com/api_response_page1.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gitlab.com/api_v4_projects,page=1,order_by=id,sort=asc,per_page=20

Loading...
View Options

swh/lister/gitlab/tests/test_lister.py

Loading...
About · Developer information · Legal