Currently the next-gen scheduler runners [1] for origins are triggered on each scheduler
machine per environment [2]. So far, they are managed manually through a cli call for
each type of visits (and eventually, once in a while per lister or forge, ...).
We need to replace those with a new module which does not exist yet, the orchestrator.
That module would be in charge of recurringly schedule origins of a given type according
to the scheduler metrics available. Deployment wise, that'd be a systemd service that
would launch it.
Its scheduling algorithm would be something like:
```
for each visit type:
- "check queue status":
if below threshold for that queue:
goto "fill-in the void state"
else
continue: "next visit type"
end
- "fill-in the void state":
for each randomly picked policy (according to ratio [3]) for that visit type:
if room in the queue and origins to schedule for that policy:
nb_tasks = ratio(task, policy) * room
push nb_tasks origins in the queue
end
end for
end for
```
[1] scheduler runners are process in charge of recurringly schedule origins per visit
type (and more if need be, e.g. lister-uuid, ...)
[2] root tmux session in:
- prod: saatchi
- staging: scheduler0
[3] Ratio could be:
```
|----------------------------------------------------------------+-----------------------------------+-------|
| visit_type | scheduling policies | ratio |
|----------------------------------------------------------------+-----------------------------------+-------|
| package-loader: archive, cran, debian, npm, nixguix, pypi, ... | already_visited_order_by_lag | 50 |
| | never_visited_oldest_update_first | 50 |
|----------------------------------------------------------------+-----------------------------------+-------|
| git, svn, hg | already_visited_order_by_lag | 49 |
| | never_visited_oldest_update_first | 49 |
| | origins_without_last_update | 2 |
|----------------------------------------------------------------+-----------------------------------+-------|
```