Add a new SWHTask class to be used as base class for swh celery tasks
ClosedPublic
Actions

Authored by douardda on Jan 8 2019, 10:50 AM.

Details

Reviewers

Group Reviewers

Commits

rDSCHb812434be7b5: Replace the TaskRouter class by a simple function
rDSCH584724d237bb: Add a new SWHTask class to be used as base class for swh celery tasks

Summary

It is meant to be used to declare swh tasks via the task decorator
instead of subclassing the (now deprecated) Task class.

It is typically used like:

from swh.scheduler.celery_backend.config import app
from swh.scheduler.tasks import SWHTask

@app.task(base=SWHTask)
def ping():
    return 'pong'

Also replace the TaskRouter class by a simple function
class-based router is a remainder of celery 3.

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

wip

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 3269
Build 4206: tox-on-jenkins	Jenkins
Build 4205: arc lint + arc unit

Event Timeline

douardda created this revision.Jan 8 2019, 10:50 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 8 2019, 10:50 AM

Harbormaster failed remote builds in B3269: Diff 2819!Jan 8 2019, 10:52 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/58/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/58/console

douardda added a parent revision: D885: Improve logging configuration in celery's setup_log_handler function and logging of the listener.Jan 8 2019, 10:55 AM

douardda added a child revision: D887: Improve cli and add 3 commands.

ardumont accepted this revision.Jan 8 2019, 11:50 AM

This revision is now accepted and ready to land.Jan 8 2019, 11:50 AM

Could we make this base class the default for all tasks registered through the @app.task decorator?

This would save us some typing on (all) the task definitions, and be a bit less error-prone.

In D886#19092, @olasd wrote:

Could we make this base class the default for all tasks registered through the @app.task decorator?

This would save us some typing on (all) the task definitions, and be a bit less error-prone.

Agreed, but I haven't find how to do that. I'll dig a bit deeper...

In D886#19337, @douardda wrote:

In D886#19092, @olasd wrote:

Could we make this base class the default for all tasks registered through the @app.task decorator?

This would save us some typing on (all) the task definitions, and be a bit less error-prone.

Agreed, but I haven't find how to do that. I'll dig a bit deeper...

It might be possible to use the task_cls argument of the Celery constructor for this. Need to check how to make this compatible with celery's pytest fixtures however...

olasd added a comment.Jan 9 2019, 5:47 PM

This comment was removed by olasd.

douardda retitled this revision from Add a new SHWTask class to be used as base class for swh celery tasks to Add a new SWHTask class to be used as base class for swh celery tasks.Jan 10 2019, 3:46 PM

rebased and typo in commit msg

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/68/ for more details.

Harbormaster completed remote builds in B3356: Diff 2882.Jan 10 2019, 3:51 PM

Closed by commit rDSCHb812434be7b5: Replace the TaskRouter class by a simple function (authored by douardda). · Explain WhyJan 10 2019, 3:53 PM

This revision was automatically updated to reflect the committed changes.