Page MenuHomeSoftware Heritage

General improvments of the indexer: Schedule indexer tasks
Closed, MigratedEdits Locked

Description

As of today, the indexer tasks are scheduled through a volatile scheduler [1] to:

  1. avoid cluttering the (scheduling) database (softwareheritage-indexer) with message too long (T991)
  1. check multiple queues prior to scheduling new ones

The indexer stack has been developed prior to the scheduler.
Now the orchestrator part seems redundant and induces quite some complexity.
If this is replaced by the actual scheduling part, it seems the point 2. can be addressed simultaneously.

So to achieve the indexer task scheduling goal, address 1. and 2.

[1] https://forge.softwareheritage.org/source/snippets/browse/master/ardumont/schedule_with_queue_length_check.py

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Oct 3 2018, 11:59 AM
ardumont created this task.
ardumont updated the task description. (Show Details)