Page MenuHomeSoftware Heritage

scheduler: Separate scheduling save code now tasks from the recurrent tasks
Closed, ResolvedPublic


It happens that the recurrent tasks get stuck. It has been worked around but the *main* issue is not yet understood.

Given the save code now expectancy "make it reliable and fast", they must be scheduled independently from the rest.
That should alleviate the current recurring pattern of the apparent "stuckness" of the save code now [1]

Implementation wise, that means having another scheduler runner separated from the current one whose concern
would be to deal with save code now only. The actual one being modified to no longer grab tasks for the save code now.

[1] It is stuck but it's not only the save code now... It's *mostly* all tasks.

Related to T3361

Event Timeline

ardumont triaged this task as Normal priority.Jun 7 2021, 4:51 PM
ardumont created this task.

Deployed first yesterday on staging (manually) to check the behavior.
Everything was fine.

So finalized the puppet part and deployed on production.

So now, in effect, we have 2 instances of existing scheduler runners [1]

One for the existing all the tasks not disabled (old behavior) which will focus only on tasks without any priority (the default).
Another for the tasks with priority (only the save code now uses those).

Closing now.

[1] well a bit more with the next-gen ones but out of scope here.