Page MenuHomeSoftware Heritage

gitlab: Add tests on pagination
ClosedPublic

Authored by ardumont on Jan 23 2021, 4:00 PM.

Details

Summary

This add one test with a test dataset to ensure pagination works as expected.

Related to T2987
Depends on D4929

Diff Detail

Repository
rDLS Listers
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18672
Build 28897: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28896: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4930 (id=17540)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..e11853e
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ojects,page=1,order_by=id,sort=asc,per_page=20} |   0
 swh/lister/gitlab/tests/test_lister.py             | 131 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 348 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_v4_projects,page=1,order_by=id,sort=asc,per_page=20} (100%)
Changes applied before test
commit e11853eb6e69cc7f38a9fec4d1dd11768911eca6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add tests on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/138/ for more details.

Refactor existing test to reuse the same mechanism as the new one

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/console

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/console

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/console

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Build is green

Patch application report for D4930 (id=17544)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..84dd616
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/143/ for more details.

anlambert added a subscriber: anlambert.

Looks good to me !

swh/lister/gitlab/tests/test_lister.py
24–25

Maybe you could add a page_url(page_id: int) in the lister to avoid duplicating URL construction in tests.
You could then write:

def test_lister_gitlab(datadir, swh_scheduler, requests_mock):
    """Gitlab lister supports full listing

    """
    instance = "gitlab.com"
    url = api_url(instance)

    response = gitlab_page_response(datadir, instance, 1)

    lister_gitlab = GitLabLister(
        swh_scheduler, url=url, instance=instance

    requests_mock.get(
        lister.page_url(1), [{"json": response}], additional_matcher=_match_request,
    )
This revision is now accepted and ready to land.Jan 25 2021, 12:28 PM
This revision was automatically updated to reflect the committed changes.
swh/lister/gitlab/tests/test_lister.py
24–25

good idea, yes.