Page MenuHomeSoftware Heritage

elasticsearch: Clean up apparent corrupted closed apache-logs indices to reclaim space
Closed, MigratedEdits Locked

Description

Apparently some old closed indices are no longer holding information [1]
Analyze further and delete those if indeed corrupted, that should free some space.

During the tests, we found the old closed indexes seems to not work correctly when they are reopened.

For example for the oldest index apache_logs-2018.07.31:

curl -s $ES_SERVER/apache_logs-2018.07.31/_stats\?pretty | jq '._all.primaries.docs'
{
  "count": 39984,
  "deleted": 0
}

A search returns nothing :

% curl $ES_SERVER/apache_logs-2018.07.31/_search\?pretty                              
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

A flush or a merge does not change anything.

The impacted indexes are from 2018-07-31 to 2020-05-11.

[1] T2787#58155

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Feb 5 2021, 9:31 AM
ardumont created this task.

Analyse and remove corrupted index script:

#!/usr/bin/env bash

# set -x
set -e

export ES_NODE="192.168.100.61:9200"

export INDEX=$1

# Open closed index
curl -s -XPOST http://$ES_NODE/$INDEX/_open\?pretty

# wait for status green
status="red"
while [ $status != "green" ]; do
    status=$(curl -s http://$ES_NODE/_cat/indices/$INDEX\?h=health)
    echo -n .
    sleep .2
done
echo

# echo "Refreshing index stats"
curl -XPOST http://$ES_NODE/$INDEX/_refresh\?pretty


# Lookup on index content index
DATA=$(curl -s http://$ES_NODE/$INDEX/_search | jq .hits.total.value)

if [ $DATA -eq 0 ]; then
    echo "Empty index $INDEX. Removing considered corrupted index..."
    curl -f -XDELETE http://$ES_NODE/$INDEX\?pretty
    echo "Index $INDEX removed"
else
    curl -s -XPOST http://$ES_NODE/$INDEX/_close\?pretty
    echo "Index $INDEX ok (closed)"
fi

echo "----"

Ran on 2018 and 2019 indices so far (got cleaned up as empty):

curl -s http://$ES_NODE/_cat/indices\?h\=index,status | grep "apache_logs-2019" | grep close | cut -f1 -d' ' | xargs -t -r -n1 ./analyse-and-destroy-corrupted-index.sh
...
ardumont changed the task status from Open to Work in Progress.Feb 5 2021, 10:43 AM
ardumont moved this task from Backlog to in-progress on the System administration board.

We took the opportunity to drop the "reindex-{logstash, swh-worker, ...}" from 2018 and before which are remnants from old migration we will never get back to.

esnode2:~% curl -s http://$ES_NODE/_cat/indices\?h\=index,status\&s\=index | grep reindex | cut -f1 -d' ' | xargs -t -n1 -i{} curl -s -XDELETE "http://$ES_NODE/{}?pretty"
...
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.