HomeSoftware Heritage

Make scheduling policy used in schedule_recurrent configurable

Description

Make scheduling policy used in schedule_recurrent configurable

Add support for a configuration option "scheduling_policy" in the config
file loaded by the 'swh scheduler schedule-recurrent' command. This
config entry allows to specify the scheduling policies used by the
schedule-recurrent tool, instead of having them hardcoded in the source
code.

A visit type policy config entry should have at least a 'weight' value
for each policy.

Default values are unchanged.

Eg.:

scheduling_policy:
  git:
    - policy: already_visited_order_by_lag
      weight: 55
      tablesample: 0.5
    - policy: never_visited_oldest_update_first
      weight: 45
      tablesample: 0.5

Note: there may not be configuration entries for all visit types, but if

a visit type policy is configured, the config entry should be complete
(in other words, the merging of the configuration with the default
values is only done at first config level).

Details