HomeSoftware Heritage

plugins: add support for scheduler's task-type declaration

Description

plugins: add support for scheduler's task-type declaration

Add a new register-task-types cli that will create missing task-type entries in the
scheduler according to:

  • only create missing task-types (do not update them), but check that the backend_name field is consistent,
  • each SWHTask-based task declared in a module listed in the 'task_modules' plugin registry field will be checked and added if needed; tasks which name start wit an underscore will not be added,
  • added task-type will have:
    • the 'type' field is derived from the task's function name (with underscores replaced with dashes),
    • the description field is the first line of that function's docstring,
    • default values as provided by the swh.lister.cli.DEFAULT_TASK_TYPE (with a simple pattern matching to have decent default values for full/incremental tasks),
    • these default values can be overloaded via the 'task_type' plugin registry entry.

For this, we had to rename all tasks names (eg. cran_lister -> list_cran).

Comes with some tests.

Details

Provenance
douarddaAuthored on Sep 3 2019, 2:39 PM
douarddaPushed on Sep 4 2019, 4:37 PM
Differential Revision
D1940: plugins: add support for scheduler's task-type declaration
Parents
rDLSe3c0ea9d9002: implement listers as plugins
Branches
Unknown
Tags
Unknown
Build Status
Buildable 7669
Build 11005: test-and-buildJenkins