Paths

Table of Contentst

Differential D2291

swh.scheduler.cli: Add `swh scheduler task-type register` cli
ClosedPublic
Actions

Authored by ardumont on Nov 18 2019, 10:24 AM.

Tags

None

Subscribers

None

Details

Reviewers

Group Reviewers

Commits

rDSCH93585727b8e0: swh.scheduler.cli: Add `swh scheduler task-type register` cli

Summary

This allows registering of workers' task types within the scheduler backend.

Those are declared in their respective setup.py's entrypoint under the section [swh.workers] .

(This is moved from swh.lister to swh.scheduler).

Test Plan

tox

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 9093
Build 13299: tox-on-jenkins	Jenkins
Build 13298: arc lint + arc unit

Event Timeline

ardumont created this revision.Nov 18 2019, 10:24 AM

ardumont added inline comments.Nov 18 2019, 10:26 AM

swh/scheduler/cli/task_type.py
77	I did not find a way to make that actually a "choice". So the current code loads all that is found through the pkg_resources mechanism.

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/307/ for more details.

Harbormaster completed remote builds in B9066: Diff 7877.Nov 18 2019, 10:26 AM

ardumont mentioned this in D2293: lister.cli: Remove task type register cli.Nov 18 2019, 10:33 AM

ardumont mentioned this in D2292: loader.package: Register loader package tasks.Nov 18 2019, 10:33 AM

ardumont mentioned this in D2294: worker: Reference the task-type registering step for loader/lister.Nov 18 2019, 10:37 AM

cli.task_type: Make the cli user able to choose the worker to load

ardumont edited the summary of this revision. (Show Details)Nov 18 2019, 10:49 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/308/ for more details.

Harbormaster completed remote builds in B9071: Diff 7882.Nov 18 2019, 10:50 AM

Add test on this use case (and refactor a bit the tests)

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/309/ for more details.

Harbormaster completed remote builds in B9072: Diff 7883.Nov 18 2019, 11:11 AM

ardumont added a child revision: D2294: worker: Reference the task-type registering step for loader/lister.Nov 18 2019, 11:27 AM

douardda requested changes to this revision.Nov 18 2019, 1:58 PM

douardda added inline comments.

swh/scheduler/__init__.py
73	May I suggest to find a better name here? I mean `load_xxx_module` makes the reader think this function will load these plugin python modules (aka import them) which it does not. It loads known swh.workers plugin declarations. In fact, I'm not even sure this function is useful. The diffs below like - for entrypoint in pkg_resources.iter_entry_points('swh.workers'): + for entrypoint in load_worker_modules().values(): looks not so useful in terms of code clarity/readability.
swh/scheduler/cli/task_type.py
35	naming again: these are not worker modules, and not even plugin (python) modules, but worker plugin descriptions. The name should be more explicit.
89	/nitpicky/ this worker_modules local variable is useless IMHO.

This revision now requires changes to proceed.Nov 18 2019, 1:58 PM

swh.scheduler.cli.task_type: Improve naming

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/310/ for more details.

Harbormaster completed remote builds in B9093: Diff 7900.Nov 18 2019, 2:21 PM

ardumont edited the summary of this revision. (Show Details)Nov 19 2019, 11:35 AM

Remove indirection and squash commits.

ardumont added inline comments.Nov 19 2019, 12:14 PM

swh/scheduler/cli/task_type.py
77	It's now a real "choice".

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/311/ for more details.

Harbormaster completed remote builds in B9112: Diff 7918.Nov 19 2019, 12:14 PM

douardda accepted this revision.Nov 19 2019, 1:48 PM

This revision is now accepted and ready to land.Nov 19 2019, 1:48 PM

Closed by commit rDSCH93585727b8e0: swh.scheduler.cli: Add `swh scheduler task-type register` cli (authored by ardumont). · Explain WhyNov 19 2019, 1:53 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSCH93585727b8e0: swh.scheduler.cli: Add `swh scheduler task-type register` cli.

ardumont mentioned this in rCDFDa74900e94946: worker: Reference the task-type registering step for loader/lister.Nov 19 2019, 1:58 PM

ardumont mentioned this in rDENVa74900e94946: worker: Reference the task-type registering step for loader/lister.Nov 26 2019, 1:35 PM

Revision Contents
Changeset List

Path

Size

requirements-test.txt

1 line

requirements.txt

1 line

swh/

scheduler/

13 lines

celery_backend/

4 lines

cli/

117 lines

tests/

10 lines

test_cli_task_type.py

125 lines

Diff 7900

requirements-test.txt

Loading...

requirements.txt

Loading...

swh/scheduler/init.py

Loading...

swh/scheduler/celery_backend/config.py

Loading...

swh/scheduler/cli/task_type.py

Loading...

swh/scheduler/tests/conftest.py

Loading...

swh/scheduler/tests/test_cli_task_type.py

Loading...