Page MenuHomeSoftware Heritage

celery: auto add tasks declared in the swh.workers entry point in task_modules

Authored by douardda on May 22 2019, 2:46 PM.



allows to declare worker tasks in a 'swh.workers' entry point. This later is expected to be a callable which returns a dict which 'tasks' key is
the module name of the celery tasks exported.

Some other 'hooks' could be provided as callable in this dictionnary,
typically db initialization stuff in listers could be handled that way.

Not sure what a decent 'API' for this shoudl look like for now, still

See D1504 for a use case implementation

Diff Detail

rDSCH Scheduling utilities
No Linters Available
No Unit Test Coverage
Build Status
Buildable 5884
Build 8062: tox-on-jenkinsJenkins
Build 8061: arc lint + arc unit

Event Timeline

douardda created this revision.May 22 2019, 2:46 PM
ardumont edited the summary of this revision. (Show Details)May 23 2019, 9:44 AM
vlorentz added inline comments.

Why not toplevel?

ardumont added inline comments.

I think the rationale is to implement a poor man's lazy policy ;)

You should mention that with this, we can drop some part of the scheduler's configuration file.
The task_modules in the yaml can be dropped.

Which is something less to worry about in the configuration part... i think ;)

ardumont added inline comments.May 23 2019, 2:45 PM

I have a little trouble reading that.

Can you explicit what the entrypoint.load() returns...

... plugging brain cables...

Nevermind, i think i grokked it:

  • entrypoint.load() returns the function registered in the For example, you dubbed it swh.lister.*.__init__.register in the other diff.
  • and then you call it, to actually access the dict the register function returns


Maybe this would be more readable?

for ...:
    worker_register_fn = entrypoint.load()

I'm fine with this btw.

Not sure what a decent 'API' for this shoudl look like for now, still experimenting.

neither do i.

vlorentz added inline comments.May 24 2019, 3:13 PM

It's a test fixture, it will always run if that file is imported

ardumont added inline comments.Jul 1 2019, 12:10 PM

heh, sounds about right indeed.

douardda updated this revision to Diff 5762.Jul 10 2019, 4:03 PM

rebased + a very small bit of doc

Now depends on D1714

douardda added inline comments.Jul 10 2019, 4:11 PM

a more explicit version is fine for me, will do


Why not toplevel?

Why toplevel?

douardda updated this revision to Diff 5765.Jul 10 2019, 4:15 PM

More explicit code for the loading of swh.workers entrypoint

in fact not just a more explcit, allows registry['tasks'] to be either a single string or a list of strings.

vlorentz added inline comments.Jul 18 2019, 11:19 AM

Because imports should always be at the toplevel unless there's a reason not to.

vlorentz requested changes to this revision.Aug 6 2019, 3:26 PM
This revision now requires changes to proceed.Aug 6 2019, 3:26 PM
douardda updated this revision to Diff 6450.Aug 28 2019, 3:54 PM

conftest: Move import statement at the top, as requested by vlorentz

vlorentz accepted this revision.Aug 30 2019, 11:10 AM
This revision is now accepted and ready to land.Aug 30 2019, 11:10 AM
douardda updated this revision to Diff 6518.Sep 2 2019, 1:35 PM

Use task_modules (a list) as registry entry

This revision was automatically updated to reflect the committed changes.