Page MenuHomeSoftware Heritage

Implement phabricator lister using the new pattern class
AbandonedPublicDraft

Authored by olasd on Dec 9 2020, 6:24 PM.

Details

Summary

Depends on D4705

Test Plan

Yeah, we need one of these...

Diff Detail

Event Timeline

Build is green

Patch application report for D4706 (id=16673)

Could not rebase; Attempt merge onto d2f4781669...

Updating d2f4781..11efbff
Fast-forward
 swh/lister/__init__.py                             |    9 +-
 swh/lister/github/__init__.py                      |    3 +-
 swh/lister/github/lister.py                        |  324 +-
 swh/lister/github/models.py                        |   17 -
 swh/lister/github/tasks.py                         |   34 +-
 .../data/https_api.github.com/empty_response.json  |    1 -
 .../data/https_api.github.com/first_response.json  |    1 -
 .../data/https_api.github.com/next_response.json   | 6702 -------------------
 .../data/https_api.github.com/repositories,since=0 | 6706 --------------------
 swh/lister/github/tests/test_lister.py             |  466 +-
 swh/lister/github/tests/test_tasks.py              |   62 +-
 swh/lister/pattern.py                              |  270 +
 swh/lister/phabricator/__init__.py                 |    3 +-
 swh/lister/phabricator/lister.py                   |  263 +-
 swh/lister/phabricator/models.py                   |   17 -
 swh/lister/phabricator/tasks.py                    |    4 +-
 swh/lister/phabricator/tests/data/__init__.py      |    0
 swh/lister/phabricator/tests/test_lister.py        |  138 +-
 swh/lister/tests/test_cli.py                       |   30 +-
 swh/lister/tests/test_pattern.py                   |  163 +
 20 files changed, 1318 insertions(+), 13895 deletions(-)
 delete mode 100644 swh/lister/github/models.py
 delete mode 100644 swh/lister/github/tests/data/https_api.github.com/empty_response.json
 delete mode 120000 swh/lister/github/tests/data/https_api.github.com/first_response.json
 delete mode 100644 swh/lister/github/tests/data/https_api.github.com/next_response.json
 delete mode 100644 swh/lister/github/tests/data/https_api.github.com/repositories,since=0
 create mode 100644 swh/lister/pattern.py
 delete mode 100644 swh/lister/phabricator/models.py
 create mode 100644 swh/lister/phabricator/tests/data/__init__.py
 create mode 100644 swh/lister/tests/test_pattern.py
Changes applied before test
commit 11efbff87fba35b205f7e103d9311b0dcffef448
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jul 22 12:05:56 2020 +0200

    Implement phabricator lister using the new pattern class

commit e420330aa5067053081f0dd5068de86e697154da
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Dec 7 18:12:15 2020 +0100

    Implement a base pattern for listers with no state storage

commit 2c7edd50e8f72b6f6f28e200aeb18a9fdfa42eed
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Dec 9 18:15:28 2020 +0100

    Reimplement the GitHub lister using the new pattern class
    
    This replaces the test data with some manually generated answers, which allows
    us to test a few more cases for instantiating the lister.
    
    This also expands test coverage to test behavior on rate-limited requests.

commit 4dab6476a0916337a38d3311b2e3b908d19f438f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Nov 25 20:40:40 2020 +0100

    Add a helper to instantiate a new-style lister from a config file
    
    This helper will be used in the task entry points.

commit 45761dcb5f782ee3045184f4dbc208ce08a4fe25
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:59:08 2020 +0200

    Hook up listers implemented with the new pattern to the CLI
    
    We stop depending on the ListerBase implementation. The main hoop we're jumping
    through is the config override mechanism in swh.lister.get_lister, as it's
    really specifc to the ListerBase `override_config` argument, which is dropped in
    pattern.Lister (in favor of explicit arguments at lister instantiation).
    
    We implement a small shim in swh.lister.pattern.Lister to give
    backwards-compatibility for the new pattern to get_lister.
    
    This generic configuration override mechanism will probably be completely
    removed when the configuration mechanism is reworked. We'll see.

commit 011928d262546c3e2af139efea854f6ea80a09e0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 10:27:57 2020 +0200

    Introduce a simpler base pattern for lister implementations.
    
    This new pattern uses the lister support features introduced in swh.scheduler to
    replace the database management done in previous iterations of the listers.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/75/ for more details.