Page MenuHomeSoftware Heritage

swh.scheduler.cli: Add `swh scheduler task-type register` cli
ClosedPublic

Authored by ardumont on Nov 18 2019, 10:24 AM.

Details

Summary

This allows registering of workers' task types within the scheduler backend.

Those are declared in their respective setup.py's entrypoint under the section [swh.workers] .

(This is moved from swh.lister to swh.scheduler).

Test Plan

tox

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Nov 18 2019, 10:24 AM
ardumont added inline comments.Nov 18 2019, 10:26 AM
swh/scheduler/cli/task_type.py
81

I did not find a way to make that actually a "choice".
So the current code loads all that is found through the pkg_resources mechanism.

ardumont updated this revision to Diff 7882.Nov 18 2019, 10:48 AM
  • cli.task_type: Make the cli user able to choose the worker to load
ardumont edited the summary of this revision. (Show Details)Nov 18 2019, 10:49 AM
ardumont updated this revision to Diff 7883.Nov 18 2019, 11:09 AM
ardumont edited the summary of this revision. (Show Details)

Add test on this use case (and refactor a bit the tests)

douardda requested changes to this revision.Nov 18 2019, 1:58 PM
douardda added inline comments.
swh/scheduler/__init__.py
74 ↗(On Diff #7883)

May I suggest to find a better name here? I mean load_xxx_module makes the reader think this function will load these plugin python modules (aka import them) which it does not.

It loads known swh.workers plugin declarations.

In fact, I'm not even sure this function is useful. The diffs below like

- for entrypoint in pkg_resources.iter_entry_points('swh.workers'):
+ for entrypoint in load_worker_modules().values():

looks not so useful in terms of code clarity/readability.

swh/scheduler/cli/task_type.py
36

naming again: these are not worker modules, and not even plugin (python) modules, but worker plugin descriptions.
The name should be more explicit.

93

/nitpicky/ this worker_modules local variable is useless IMHO.

This revision now requires changes to proceed.Nov 18 2019, 1:58 PM
ardumont updated this revision to Diff 7900.Nov 18 2019, 2:19 PM
  • swh.scheduler.cli.task_type: Improve naming
ardumont edited the summary of this revision. (Show Details)Nov 19 2019, 11:35 AM
ardumont updated this revision to Diff 7918.Nov 19 2019, 12:12 PM

Remove indirection and squash commits.

ardumont added inline comments.Nov 19 2019, 12:14 PM
swh/scheduler/cli/task_type.py
81

It's now a real "choice".

douardda accepted this revision.Nov 19 2019, 1:48 PM
This revision is now accepted and ready to land.Nov 19 2019, 1:48 PM