Page MenuHomeSoftware Heritage
Differential D4930

gitlab: Add tests on pagination
ClosedPublic
Actions

Authored by ardumont on Jan 23 2021, 4:00 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API
Commits
rDLS84dd616ab641: gitlab: Add test on pagination
Summary

This add one test with a test dataset to ensure pagination works as expected.

Related to T2987
Depends on D4929

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jan 23 2021, 4:00 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 23 2021, 4:00 PM
swh-public-ci added a comment.Jan 23 2021, 4:06 PM

Build is green

Patch application report for D4930 (id=17540)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..e11853e
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ojects,page=1,order_by=id,sort=asc,per_page=20} |   0
 swh/lister/gitlab/tests/test_lister.py             | 131 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 348 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_v4_projects,page=1,order_by=id,sort=asc,per_page=20} (100%)
Changes applied before test
commit e11853eb6e69cc7f38a9fec4d1dd11768911eca6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add tests on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/138/ for more details.

Harbormaster completed remote builds in B18672: Diff 17540.Jan 23 2021, 4:06 PM
ardumont requested review of this revision.Jan 23 2021, 4:06 PM
ardumont updated this revision to Diff 17541.Jan 23 2021, 4:13 PM

Refactor existing test to reuse the same mechanism as the new one

swh-public-ci added a comment.Jan 23 2021, 4:16 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/139/console

Harbormaster failed remote builds in B18673: Diff 17541!Jan 23 2021, 4:16 PM
swh-public-ci added a comment.Jan 23 2021, 4:29 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/140/console

swh-public-ci added a comment.Jan 23 2021, 4:39 PM

Build has FAILED

Patch application report for D4930 (id=17541)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..fd0eb63
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit fd0eb63a1dcca676482a1bd4d90d8d2b3171136d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination

commit 4611d7d75123ffd6ed2517a677f2000b53939887
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/141/console

ardumont edited the test plan for this revision. (Show Details)Jan 25 2021, 8:46 AM
ardumont added a task: T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API.
ardumont edited the summary of this revision. (Show Details)Jan 25 2021, 8:52 AM
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont added a parent revision: D4929: gitlab: Port to the new lister api.
ardumont added a child revision: D4932: gitlab: Add test on incremental implementation.Jan 25 2021, 8:53 AM
ardumont mentioned this in D4929: gitlab: Port to the new lister api.Jan 25 2021, 8:55 AM
ardumont updated this revision to Diff 17544.Jan 25 2021, 8:56 AM

Rebase

ardumont edited the summary of this revision. (Show Details)Jan 25 2021, 8:56 AM
swh-public-ci added a comment.Jan 25 2021, 8:59 AM

Build is green

Patch application report for D4930 (id=17544)

Could not rebase; Attempt merge onto ff232f0d91...

Updating ff232f0..84dd616
Fast-forward
 swh/lister/gitlab/__init__.py                      |   4 +-
 swh/lister/gitlab/lister.py                        | 253 +++++++++++++++------
 swh/lister/gitlab/models.py                        |  18 --
 swh/lister/gitlab/tests/conftest.py                |  26 ---
 .../https_gite.lirmm.fr/api_response_page1.json    |  42 ++++
 .../https_gite.lirmm.fr/api_response_page2.json    |  42 ++++
 .../data/https_gitlab.com/api_empty_response.json  |   1 -
 .../tests/data/https_gitlab.com/api_response.json  |   1 -
 ...ort=asc,per_page=20 => api_response_page1.json} |   0
 swh/lister/gitlab/tests/test_lister.py             | 139 +++++++----
 swh/lister/tests/test_cli.py                       |   1 -
 11 files changed, 356 insertions(+), 171 deletions(-)
 delete mode 100644 swh/lister/gitlab/models.py
 delete mode 100644 swh/lister/gitlab/tests/conftest.py
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json
 create mode 100644 swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json
 delete mode 100644 swh/lister/gitlab/tests/data/https_gitlab.com/api_empty_response.json
 delete mode 120000 swh/lister/gitlab/tests/data/https_gitlab.com/api_response.json
 rename swh/lister/gitlab/tests/data/https_gitlab.com/{api_v4__projects,page=0,order_by=id,sort=asc,per_page=20 => api_response_page1.json} (100%)
Changes applied before test
commit 84dd616ab641ff91af33fdea46cfe7f91b4a773b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:51:18 2021 +0100

    gitlab: Add test on pagination
    
    Related to T2987

commit 1390a513f2e53487dc8cf0aa36a959be4e4c1eef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Jan 23 15:18:20 2021 +0100

    gitlab: Port to the new lister api
    
    Related to T2987

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B18678: Diff 17544.Jan 25 2021, 8:59 AM
anlambert accepted this revision.Jan 25 2021, 12:28 PM
anlambert added a subscriber: anlambert.

Looks good to me !

swh/lister/gitlab/tests/test_lister.py
24–25

Maybe you could add a page_url(page_id: int) in the lister to avoid duplicating URL construction in tests.
You could then write:

def test_lister_gitlab(datadir, swh_scheduler, requests_mock):
    """Gitlab lister supports full listing

    """
    instance = "gitlab.com"
    url = api_url(instance)

    response = gitlab_page_response(datadir, instance, 1)

    lister_gitlab = GitLabLister(
        swh_scheduler, url=url, instance=instance

    requests_mock.get(
        lister.page_url(1), [{"json": response}], additional_matcher=_match_request,
    )
This revision is now accepted and ready to land.Jan 25 2021, 12:28 PM
Closed by commit rDLS84dd616ab641: gitlab: Add test on pagination (authored by ardumont). · Explain WhyJan 25 2021, 12:54 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS84dd616ab641: gitlab: Add test on pagination.
ardumont added inline comments.Jan 25 2021, 1:22 PM
swh/lister/gitlab/tests/test_lister.py
24–25

good idea, yes.

Revision Contents
Changeset List

PathSize
swh/
lister/
gitlab/
tests/
data/
https_gite.lirmm.fr/
42 lines
42 lines
https_gitlab.com/
api_response_page1.json
api_v4_projects,page=1,order_by=id,sort=asc,per_page=20
103 lines

Diff 17554

View Options

swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page1.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gite.lirmm.fr/api_response_page2.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gitlab.com/api_response_page1.json

Loading...
View Options

swh/lister/gitlab/tests/data/https_gitlab.com/api_v4_projects,page=1,order_by=id,sort=asc,per_page=20

Loading...
View Options

swh/lister/gitlab/tests/test_lister.py

Loading...
About · Developer information · Legal