This can be used for listers that always iterate on the full upstream API, like Phabricator or PyPI.
Depends on D4700. (sort of; at least if you don't want merge conflicts)
Differential D4705
Implement a base pattern for listers with no state storage Authored by olasd on Dec 9 2020, 6:23 PM. Tags None Subscribers None
Details
This can be used for listers that always iterate on the full upstream API, like Phabricator or PyPI. Depends on D4700. (sort of; at least if you don't want merge conflicts) new tox tests introduced
Diff Detail
Event TimelineComment Actions Build is green Patch application report for D4705 (id=16672)Could not rebase; Attempt merge onto d2f4781669... Updating d2f4781..e420330 Fast-forward swh/lister/__init__.py | 9 +- swh/lister/github/__init__.py | 3 +- swh/lister/github/lister.py | 324 +- swh/lister/github/models.py | 17 - swh/lister/github/tasks.py | 34 +- .../data/https_api.github.com/empty_response.json | 1 - .../data/https_api.github.com/first_response.json | 1 - .../data/https_api.github.com/next_response.json | 6702 ------------------- .../data/https_api.github.com/repositories,since=0 | 6706 -------------------- swh/lister/github/tests/test_lister.py | 466 +- swh/lister/github/tests/test_tasks.py | 62 +- swh/lister/pattern.py | 270 + swh/lister/tests/test_cli.py | 18 +- swh/lister/tests/test_pattern.py | 163 + 14 files changed, 1165 insertions(+), 13611 deletions(-) delete mode 100644 swh/lister/github/models.py delete mode 100644 swh/lister/github/tests/data/https_api.github.com/empty_response.json delete mode 120000 swh/lister/github/tests/data/https_api.github.com/first_response.json delete mode 100644 swh/lister/github/tests/data/https_api.github.com/next_response.json delete mode 100644 swh/lister/github/tests/data/https_api.github.com/repositories,since=0 create mode 100644 swh/lister/pattern.py create mode 100644 swh/lister/tests/test_pattern.py Changes applied before testcommit e420330aa5067053081f0dd5068de86e697154da
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date: Mon Dec 7 18:12:15 2020 +0100
Implement a base pattern for listers with no state storage
commit 2c7edd50e8f72b6f6f28e200aeb18a9fdfa42eed
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date: Wed Dec 9 18:15:28 2020 +0100
Reimplement the GitHub lister using the new pattern class
This replaces the test data with some manually generated answers, which allows
us to test a few more cases for instantiating the lister.
This also expands test coverage to test behavior on rate-limited requests.
commit 4dab6476a0916337a38d3311b2e3b908d19f438f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date: Wed Nov 25 20:40:40 2020 +0100
Add a helper to instantiate a new-style lister from a config file
This helper will be used in the task entry points.
commit 45761dcb5f782ee3045184f4dbc208ce08a4fe25
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date: Thu Jul 16 11:59:08 2020 +0200
Hook up listers implemented with the new pattern to the CLI
We stop depending on the ListerBase implementation. The main hoop we're jumping
through is the config override mechanism in swh.lister.get_lister, as it's
really specifc to the ListerBase `override_config` argument, which is dropped in
pattern.Lister (in favor of explicit arguments at lister instantiation).
We implement a small shim in swh.lister.pattern.Lister to give
backwards-compatibility for the new pattern to get_lister.
This generic configuration override mechanism will probably be completely
removed when the configuration mechanism is reworked. We'll see.
commit 011928d262546c3e2af139efea854f6ea80a09e0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date: Mon Jul 6 10:27:57 2020 +0200
Introduce a simpler base pattern for lister implementations.
This new pattern uses the lister support features introduced in swh.scheduler to
replace the database management done in previous iterations of the listers.See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/74/ for more details. |