Currently the next-gen scheduler runners in charge of scheduling recurring visits for[1] for origins are triggered on each scheduler
origins are triggered on the scheduler machines per environment [1].2]. So far, Tthey are managed manually through a cli call for
individually and manually through a cli call for each type of visits (and eventually per
listereach type of visits (and eventually, once in a while per lister or forge, ...).
We need to replace those with a new module which does not exist yet, the orchestrator.
That module would be in charge of regularlyrecurringly schedule origins of a given type according
to the scheduler metrics available. Deployment wise, that'd be a systemd service that
would launch it.
Its scheduling algorightthm would followbe something along those lines:ng like:
```
for each visit type:
- queue filled-in: monitored, when under a given threshold event is reached,- "check queue status":
this triggers a fill-in the void state. if below threshold for that queue:
goto "fill-in the void state"
else
continue: "next visit type"
end
- - "fill-in the void state:":
for each (randomly picked?) policy (according to ratio [13]) for that visit type:
- fetch if room in the queue and origins andto schedule for that policy:
- continue until threshold is reached or policy has no results schedule origins: push messages in the queue
elif room in the queue and no more origins to schedule for that policy:
continue: "next policy"
else ("no more room in the queue"):
continue: "next visit type"
end
- when threshold is reached, goto "queue filled-in"end for
end for
```
[1] scheduler runners are process in charge of recurringly schedule origins per visit
type (and more if need be, e.g. lister-uuid, ...)
[12] root tmux session in:
- prod: saatchi
- staging: scheduler0
[3] Ratio could be:
```
|----------------------------------------------------------------+-----------------------------------+-------|
| visit_type | scheduling policies | ratio |
|----------------------------------------------------------------+-----------------------------------+-------|
| package-loader: archive, cran, debian, npm, nixguix, pypi, ... | already_visited_order_by_lag | 50 |
| | never_visited_oldest_update_first | 50 |
|----------------------------------------------------------------+-----------------------------------+-------|
| git, svn, hg | already_visited_order_by_lag | 49 |
| | never_visited_oldest_update_first | 49 |
| | origins_without_last_update | 2 |
|----------------------------------------------------------------+-----------------------------------+-------|
```
[1] root tmux session in:
- prod: saatchi
- staging: scheduler0