Page MenuHomeSoftware Heritage

Indexers: Make orchestrators use swh-scheduler for scheduling
Closed, MigratedEdits Locked

Description

At the moment, the orchestrator 'schedule' directly tasks...

Event Timeline

ardumont triaged this task as Normal priority.Oct 3 2018, 12:04 PM
ardumont created this task.

@vlorentz @olasd Beware though, the orchestrator is supposed to per indexer (as in its configuration, e.g. P235):

  • receive data (input)
  • split it (according to indexer configuration)
  • optionally filter it (according to indexer configuration)
  • forward the result (objects to index) to indexer

That's why there is still analysis needed.
I'm not entirely sure we can remove it.

A more middleground approach could be to change its implementation to remove the immediate message sending to the indexers ("immediate scheduling" of sort).
Instead, lift the scheduling api to add new tasks in the scheduler.
And let the scheduler do its job (schedule ;)

My plan is to move the logic from the orchestrator into the indexers themselves

My plan is to move the logic from the orchestrator into the indexers themselves

I do not think that's a reasonable approach (it's currently running in production and it works, i'd like to keep it that way ;).

The idea of the intermediate layer (orchestrator) is to be flexible in configuration without changing runtime code.
We want to keep that configuration flexibility.

Currently, if we want to add or remove an indexer, we can just change the configuration and no new deployment is needed (well, except for new indexer code, but the rest stands, we just need to change the right orchestrator's configuration).

Indexers can be made configurable as well

Indexers can be made configurable as well

Yes and they are already, only for what they are supposed to do, indexing.
With what you propose, I'm afraid to lose the existing separation of concern logic.

vlorentz added a subscriber: vlorentz.
ardumont renamed this task from Indexers: Analysis further the possiblity to remove the orchestrator layer to the benefit of the scheduler to Indexers: Make orchestrators use swh-scheduler for scheduling.Oct 26 2018, 8:05 PM
ardumont updated the task description. (Show Details)

So this task will be solved by D606, right?

ardumont claimed this task.

Yes, it should.

So in the end, we will remove the orchestrator anyway.
That will simplify the indexer scheduling (which could not really happen easily in the current state).

So opening a new task for it.