Change Details

During the rolling restart of the cluster 2 disks failures has crashed esnode1 and avoid the cluster to recover. [Copied from a comment] Short term plan : [x] Remove old systemlogs indexes older than 1year to start, but we can go to 3 months if necessary [x] reactivate the shard allocation to have 1 replica for all the shards in case of a second node failure [x] Launch a long smartcl test on all the disks of each esnode* server [] Contact DELL support to proceed to the replacement of the 2 failing disks (under warranty(?)) [1] [x] Try to recover the 16 red indexes if possible, if not, delete them as they are not critical Middle term: [] Reconfigure sentry to use its local kafka instance instead of the esnode* kafka cluster [] Cleanup the esnode* kafka/zookeeper instances and reclaim the 2To disk reserved for the journal [] Add a new datadir on elasticsearch using the new available disk [] Add smartctl monitoring to detect disk failure as soon as possible [1] sdb serial : K5GJBLTA / sdc serial : K5GV9REA

During the rolling restart of the cluster 2 disks failures has crashed esnode1 and avoid the cluster to recover. [Copied from a comment] Short term plan : [x] Remove old systemlogs indexes older than 1year to start, but we can go to 3 months if necessary [x] reactivate the shard allocation to have 1 replica for all the shards in case of a second node failure [x] Launch a long smartcl test on all the disks of each esnode* server [] Contact DELL support to proceed to the replacement of the 2 failing disks (under warranty(?)) [1] [x] Try to recover the 16 red indexes if possible, if not, delete them as they are not critical Middle term: [x] Reconfigure sentry to use its local kafka instance instead of the esnode* kafka cluster (thank olasd) [] Cleanup the esnode* kafka/zookeeper instances and reclaim the 2To disk reserved for the journal [] Add a new datadir on elasticsearch using the new available disk [] Add smartctl monitoring to detect disk failure as soon as possible [1] sdb serial : K5GJBLTA / sdc serial : K5GV9REA

During the rolling restart of the cluster 2 disks failures has crashed esnode1 and avoid the cluster to recover. [Copied from a comment] Short term plan : [x] Remove old systemlogs indexes older than 1year to start, but we can go to 3 months if necessary [x] reactivate the shard allocation to have 1 replica for all the shards in case of a second node failure [x] Launch a long smartcl test on all the disks of each esnode* server [] Contact DELL support to proceed to the replacement of the 2 failing disks (under warranty(?)) [1] [x] Try to recover the 16 red indexes if possible, if not, delete them as they are not critical Middle term: [x] Reconfigure sentry to use its local kafka instance instead of the esnode* kafka cluster (thank olasd) [] Cleanup the esnode* kafka/zookeeper instances and reclaim the 2To disk reserved for the journal [] Add a new datadir on elasticsearch using the new available disk [] Add smartctl monitoring to detect disk failure as soon as possible [1] sdb serial : K5GJBLTA / sdc serial : K5GV9REA