Page MenuHomeSoftware Heritage

celery: auto add tasks declared in the swh.workers entry point in task_modules
Needs ReviewPublic

Authored by douardda on Wed, May 22, 2:46 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

allows to declare worker tasks in a 'swh.workers' entry point. This later is expected to be a callable which returns a dict which 'tasks' key is
the module name of the celery tasks exported.

Some other 'hooks' could be provided as callable in this dictionnary,
typically db initialization stuff in listers could be handled that way.

Not sure what a decent 'API' for this shoudl look like for now, still
experimenting.

See D1504 for a use case implementation

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
wip
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5884
Build 8062: tox-on-jenkinsJenkins
Build 8061: arc lint + arc unit

Event Timeline

douardda created this revision.Wed, May 22, 2:46 PM
ardumont edited the summary of this revision. (Show Details)Thu, May 23, 9:44 AM
vlorentz added inline comments.
swh/scheduler/tests/conftest.py
40

Why not toplevel?

ardumont added inline comments.
swh/scheduler/tests/conftest.py
40

I think the rationale is to implement a poor man's lazy policy ;)

You should mention that with this, we can drop some part of the scheduler's configuration file.
The task_modules in the yaml can be dropped.

Which is something less to worry about in the configuration part... i think ;)

ardumont added inline comments.Thu, May 23, 2:45 PM
swh/scheduler/celery_backend/config.py
204

I have a little trouble reading that.

Can you explicit what the entrypoint.load() returns...

... plugging brain cables...

Nevermind, i think i grokked it:

  • entrypoint.load() returns the function registered in the setup.py. For example, you dubbed it swh.lister.*.__init__.register in the other diff.
  • and then you call it, to actually access the dict the register function returns

Ok.

Maybe this would be more readable?

for ...:
    worker_register_fn = entrypoint.load()
    CONFIG['task_modules'].append(worker_register_fn()['tasks])

I'm fine with this btw.

Not sure what a decent 'API' for this shoudl look like for now, still experimenting.

neither do i.

vlorentz added inline comments.Fri, May 24, 3:13 PM
swh/scheduler/tests/conftest.py
40

It's a test fixture, it will always run if that file is imported