HomeSoftware Heritage

scheduling: Add direct task to celery using queue length check

Description

scheduling: Add direct task to celery using queue length check

For real huge data source (googlecode svn, indexers), using the
scheduler db is a no go (for now).

This leverages the same idea behind
swh.scheduler.backend_celery.runner without using the scheduler db.

This will permit to improve the rescheduling of T879 (googlecode svn).

Related T864
Related T879

Details

Provenance
ardumontAuthored on Dec 11 2017, 2:01 PM
ardumontPushed on Dec 11 2017, 5:00 PM
Parents
rDSNIP5b55de67ddb6: Add some more filtering logs
Branches
Unknown
Tags
Unknown