General improvements to the scheduler
Open, HighPublic
Actions

Description

After a whiteboarding session with @zack related to T720, we've identified some of the pain points of the scheduler, as well as some feature additions that are going to be useful in the long run.

This is a meta-task to collect all the improvements we have discussed

Related Objects
Search...

Status	Assigned	Task
Open	None	T533 Allow loaders to register partial state (meta task)
Open	olasd	T723 General improvements to the scheduler
Resolved	olasd	T720 (Re)Scheduling oneshot tasks automatically
Resolved	olasd	T537 Update the task scheduler and the task event listener to use the new partial status
Resolved	None	T536 Enrich task return status to register partial completions
Open	None	T535 Register partial task state in scheduler database
Open	olasd	T724 Move scheduler task run logging to a more permanent location
Resolved	ardumont	T986 Scheduler: Automate completed oneshot or disabled recurring tasks archival
Work in Progress	ftigeot	T792 Make the elasticsearch logging cluster actually a cluster
Work in Progress	None	T793 Move elasticsearch log cluster configuration inside puppet
Resolved	ftigeot	T945 Separate system logs from application logs
Resolved	ftigeot	T977 Delete old system log data from the Elasticsearch cluster
Resolved	ftigeot	T983 Make logstash on banco store documents on Elasticsearch version 6.x nodes
Resolved	ftigeot	T990 Tune index parameters
Resolved	ftigeot	T1000 Reindex old data on banco to put it into swh_worker indexes
Open	olasd	T1017 Estimate for Kafka cluster specifications
Resolved	ardumont	T1023 scheduler: tasks archival: Test run on test elasticsearch cluster
Resolved	ardumont	T1071 Scheduler: Archive existing completed oneshot and disabled recurring tasks
Resolved	ardumont	T725 Scheduler API server
Resolved	olasd	T726 Add "queue shaping" to the scheduler