Page MenuHomeSoftware Heritage

gitlab: Add tests on pagination
ClosedPublic

Authored by ardumont on Jan 23 2021, 4:00 PM.

Details

Summary

This add one test with a test dataset to ensure pagination works as expected.

Related to T2987
Depends on D4929

Diff Detail

Event Timeline

Build is green

Patch application report for D4930 (id=17540)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..e11853e
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ojects,page=1,order_by=id,sort=asc,per_page=20} |   0
 swh/lister/gitlab/tests/test_lister.py             | 131 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 348 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_v4_projects,page=1,order_by=id,sort=asc,per_page=20} (100%)
Changes applied before test
commit e11853eb6e69cc7f38a9fec4d1dd11768911eca6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add tests on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/138/ for more details.

Refactor existing test to reuse the same mechanism as the new one

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/console

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/console

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/console

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Build is green

Patch application report for D4930 (id=17544)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..84dd616
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/143/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me !

swh/lister/gitlab/tests/test_lister.py
24

Maybe you could add a page_url(page_id: int) in the lister to avoid duplicating URL construction in tests.
You could then write:

def test_lister_gitlab(datadir, swh_scheduler, requests_mock):
    """Gitlab lister supports full listing

    """
    instance = "gitlab.com"
    url = api_url(instance)

    response = gitlab_page_response(datadir, instance, 1)

    lister_gitlab = GitLabLister(
        swh_scheduler, url=url, instance=instance

    requests_mock.get(
        lister.page_url(1), [{"json": response}], additional_matcher=_match_request,
    )
This revision is now accepted and ready to land.Jan 25 2021, 12:28 PM
This revision was automatically updated to reflect the committed changes.
swh/lister/gitlab/tests/test_lister.py
24

good idea, yes.