Page MenuHomeSoftware Heritage

elk: automatically limit log retention
Closed, MigratedEdits Locked


The index retention should be automatically managed to limit the number of opened indexes.

There are 2 types of indexes on the cluster:

  • functional logs indexes (task-.* swh_workers-.*
  • system log indexes (apache_logs-.* systemlogs-.*)

The rules can be :

  • both types of indexes: close indexes older than 6 months
  • system logs: delete indexes older than 12 months

Event Timeline

vsellier triaged this task as Normal priority.Apr 8 2021, 4:30 PM
vsellier created this task.

A script is regurarly executed to close the oldest indexes (30days) : P1004
It should be added on puppet and scheduled in a cron

ardumont changed the task status from Open to Work in Progress.Apr 12 2021, 3:56 PM
ardumont edited projects, added System administration; removed System administrators.
ardumont moved this task from Backlog to in-progress on the System administration board.

^ Deployed

root@logstash0:~# cat /etc/cron.d/puppet-elasticsearch
# Managed by puppet (module profile::cron), manual changes will be lost

# Cron snippet elasticsearch-close-index
30 5 * * * root chronic sh -c '/usr/local/bin/ --host --host --host --timeout 1200'
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.