Page MenuHomeSoftware Heritage
Differential D3526

Hook up listers implemented with the new pattern to the CLI
ClosedPublic
Actions

Authored by olasd on Jul 16 2020, 12:24 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDLS9e083c1eead6: Introduce a simpler base pattern for lister implementations.
Summary

We stop depending on the ListerBase implementation. The main hoop we're jumping
through is the config override mechanism in swh.lister.get_lister, as it's
really specifc to the ListerBase override_config argument, which is dropped in
pattern.Lister (in favor of overriding a load_config method).

This generic configuration override mechanism will probably be completely
superseded when all lister instances are reimplemented. We'll see.

Depends on D3425.

Test Plan

will really be used when we actually implement some listers on top of this

Diff Detail

Event Timeline

olasd created this revision.Jul 16 2020, 12:24 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 16 2020, 12:24 PM
olasd added a child revision: D3527: Reimplement the GitHub lister using the new pattern class.Jul 16 2020, 12:28 PM
swh-public-ci added a comment.Jul 16 2020, 12:32 PM

Build is green

Patch application report for D3526 (id=12453)

Could not rebase; Attempt merge onto 5f1fbbe8a4...

Updating 5f1fbbe..91ccd2b
Fast-forward
 MANIFEST.in                       |   3 +-
 conftest.py                       |  19 +++
 requirements-swh.txt              |   2 +-
 swh/lister/__init__.py            |  18 ++-
 swh/lister/core/tests/conftest.py |   4 +-
 swh/lister/pattern.py             | 236 ++++++++++++++++++++++++++++++++++++++
 swh/lister/tests/test_cli.py      |   9 +-
 swh/lister/tests/test_pattern.py  |  93 +++++++++++++++
 8 files changed, 370 insertions(+), 14 deletions(-)
 create mode 100644 swh/lister/pattern.py
 create mode 100644 swh/lister/tests/test_pattern.py
Changes applied before test
commit 91ccd2bd25682e8b1d2f6e0e5ea706fddd1f5abe
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:59:08 2020 +0200

    Hook up listers implemented with the new pattern to the CLI
    
    We stop depending on the ListerBase implementation. The main hoop we're jumping
    through is the config override mechanism in swh.lister.get_lister, as it's
    really specifc to the ListerBase `override_config` argument, which is dropped in
    pattern.Lister (in favor of overriding a `load_config` method).
    
    This generic configuration override mechanism will probably be completely
    superseded when all lister instances are reimplemented. We'll see.

commit 4d9db6e09f500285f2f1a484aeb6c11be674394c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 10:27:57 2020 +0200

    Intoduce a simpler base pattern for lister implementations.
    
    This new pattern uses the lister support features introduced in swh.scheduler to
    replace the database management done in previous iterations of the listers.

commit 211f4610df2006e939bb3f6e8f2cb7b87151dfba
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:45:58 2020 +0200

    Move get_scheduler monkeypatching into an explicit pytest fixture
    
    This allows us to actually run the lister instantiation code instead of relying
    on the underlying structure of the lister object. In turn, this allows future
    listers to use the scheduler right in their __init__.

commit d0c1df65f1f4b05cd9e766ee3916d25182ecdb25
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 10:27:54 2020 +0200

    Only include relevant data files in MANIFEST.in
    
    The previous include would pull all .mypy_cache directories too, which are quite
    large as they include the SQLAlchemy stubs.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/23/ for more details.

Harbormaster completed remote builds in B13675: Diff 12453.Jul 16 2020, 12:32 PM
ardumont accepted this revision.Jul 16 2020, 1:22 PM
ardumont added a subscriber: ardumont.

This generic configuration override mechanism will probably be completely
superseded when all lister instances are reimplemented. We'll see.

Hopefully yes ;)

This revision is now accepted and ready to land.Jul 16 2020, 1:22 PM
olasd updated this revision to Diff 12645.Jul 22 2020, 12:46 PM

Rebase on top of updated D3425

swh-public-ci added a comment.Jul 22 2020, 12:57 PM

Build is green

Patch application report for D3526 (id=12645)

Could not rebase; Attempt merge onto 211f4610df...

Updating 211f461..964e097
Fast-forward
 conftest.py                      |   7 +-
 requirements-swh.txt             |   2 +-
 swh/lister/__init__.py           |  18 ++-
 swh/lister/pattern.py            | 238 +++++++++++++++++++++++++++++++++++++++
 swh/lister/tests/test_cli.py     |   7 +-
 swh/lister/tests/test_pattern.py |  93 +++++++++++++++
 6 files changed, 354 insertions(+), 11 deletions(-)
 create mode 100644 swh/lister/pattern.py
 create mode 100644 swh/lister/tests/test_pattern.py
Changes applied before test
commit 964e09764524ba4f4dfafb15ecbf72eaa927cfdb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:59:08 2020 +0200

    Hook up listers implemented with the new pattern to the CLI
    
    We stop depending on the ListerBase implementation. The main hoop we're jumping
    through is the config override mechanism in swh.lister.get_lister, as it's
    really specifc to the ListerBase `override_config` argument, which is dropped in
    pattern.Lister (in favor of overriding a `load_config` method).
    
    This generic configuration override mechanism will probably be completely
    superseded when all lister instances are reimplemented. We'll see.

commit 7ab4f69692af27b47a41566fc6db7aeeba4dda66
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 10:27:57 2020 +0200

    Introduce a simpler base pattern for lister implementations.
    
    This new pattern uses the lister support features introduced in swh.scheduler to
    replace the database management done in previous iterations of the listers.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/26/ for more details.

Harbormaster completed remote builds in B13870: Diff 12645.Jul 22 2020, 12:57 PM
olasd updated this revision to Diff 16314.Nov 25 2020, 4:36 PM

Rebase

swh-public-ci added a comment.Nov 25 2020, 4:37 PM

Build has FAILED

Patch application report for D3526 (id=16314)

Could not rebase; Attempt merge onto d2f4781669...

Updating d2f4781..f92a392
Fast-forward
 swh/lister/__init__.py           |   9 +-
 swh/lister/pattern.py            | 249 +++++++++++++++++++++++++++++++++++++++
 swh/lister/tests/test_cli.py     |   7 +-
 swh/lister/tests/test_pattern.py |  98 +++++++++++++++
 4 files changed, 356 insertions(+), 7 deletions(-)
 create mode 100644 swh/lister/pattern.py
 create mode 100644 swh/lister/tests/test_pattern.py
Changes applied before test
commit f92a3925fcd104546d1d5055a6bb28790cac1016
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:59:08 2020 +0200

    Hook up listers implemented with the new pattern to the CLI
    
    We stop depending on the ListerBase implementation. The main hoop we're jumping
    through is the config override mechanism in swh.lister.get_lister, as it's
    really specifc to the ListerBase `override_config` argument, which is dropped in
    pattern.Lister (in favor of explicit arguments at lister instantiation).
    
    We implement a small shim to give backwards-compatibility for the new pattern to
    get_lister.
    
    This generic configuration override mechanism will probably be completely
    superseded when all lister instances are reimplemented. We'll see.

commit 6eafbe5c764725c7ee46c33bc984fe42da28a8a1
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 10:27:57 2020 +0200

    Introduce a simpler base pattern for lister implementations.
    
    This new pattern uses the lister support features introduced in swh.scheduler to
    replace the database management done in previous iterations of the listers.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/69/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/69/console

Harbormaster failed remote builds in B17506: Diff 16314!Nov 25 2020, 4:37 PM
olasd updated this revision to Diff 16664.Dec 9 2020, 5:51 PM

Implement this again using a classmethod

olasd added a child revision: D4700: Add a helper to instantiate a new-style lister from a config file.Dec 9 2020, 5:52 PM
swh-public-ci added a comment.Dec 9 2020, 5:56 PM

Build is green

Patch application report for D3526 (id=16664)

Could not rebase; Attempt merge onto d2f4781669...

Updating d2f4781..45761dc
Fast-forward
 swh/lister/__init__.py           |   9 +-
 swh/lister/pattern.py            | 248 +++++++++++++++++++++++++++++++++++++++
 swh/lister/tests/test_cli.py     |  18 ++-
 swh/lister/tests/test_pattern.py | 113 ++++++++++++++++++
 4 files changed, 375 insertions(+), 13 deletions(-)
 create mode 100644 swh/lister/pattern.py
 create mode 100644 swh/lister/tests/test_pattern.py
Changes applied before test
commit 45761dcb5f782ee3045184f4dbc208ce08a4fe25
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jul 16 11:59:08 2020 +0200

    Hook up listers implemented with the new pattern to the CLI
    
    We stop depending on the ListerBase implementation. The main hoop we're jumping
    through is the config override mechanism in swh.lister.get_lister, as it's
    really specifc to the ListerBase `override_config` argument, which is dropped in
    pattern.Lister (in favor of explicit arguments at lister instantiation).
    
    We implement a small shim in swh.lister.pattern.Lister to give
    backwards-compatibility for the new pattern to get_lister.
    
    This generic configuration override mechanism will probably be completely
    removed when the configuration mechanism is reworked. We'll see.

commit 011928d262546c3e2af139efea854f6ea80a09e0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Mon Jul 6 10:27:57 2020 +0200

    Introduce a simpler base pattern for lister implementations.
    
    This new pattern uses the lister support features introduced in swh.scheduler to
    replace the database management done in previous iterations of the listers.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/71/ for more details.

Harbormaster completed remote builds in B17838: Diff 16664.Dec 9 2020, 5:56 PM
olasd removed a child revision: D3527: Reimplement the GitHub lister using the new pattern class.Dec 9 2020, 6:17 PM
olasd closed this revision.Jan 11 2021, 3:53 PM
olasd added a commit: rDLS9e083c1eead6: Introduce a simpler base pattern for lister implementations..

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
lister/
9 lines86%
Loading...
26 lines82%
Loading...
tests/
18 lines100%
Loading...

Diff 16664

View Options

swh/lister/__init__.py

Loading...
View Options

swh/lister/pattern.py

Loading...
View Options

swh/lister/tests/test_cli.py

Loading...
About · Developer information · Legal