Kibana was updated in early 2020, and the existing config is no longer read.
20:09 <+olasd> (had to change the configured config index from .kibana-6 to .kibana-7) 20:16 <+olasd> (looks like the config is therefore blank, which is... annoying) 20:17 <+olasd> (i'll stop poking at it now, please make a task to migrate/restore the old config?)
Kibana uses an elasticsearch index to store its settings.
The concrete implementation is as follows:
When upgrading, kibana migrates the contents of .kibana_XXX to .kibana_XXX+1 then atomically updates the alias.
After restoring the index and alias to the previously working state, and restarting the kibana service, the following logs have appeared:
Sep 09 11:33:26 kibana0 kibana[2133]: {"type":"log","@timestamp":"2020-09-09T11:33:26Z","tags":["info","savedobjects-service"],"pid":2133,"message":"Starting saved objects migrations"}
Sep 09 11:33:27 kibana0 kibana[2133]: {"type":"log","@timestamp":"2020-09-09T11:33:27Z","tags":["info","savedobjects-service"],"pid":2133,"message":"Creating index .kibana_9."}
Sep 09 11:33:28 kibana0 kibana[2133]: {"type":"log","@timestamp":"2020-09-09T11:33:28Z","tags":["info","savedobjects-service"],"pid":2133,"message":"Migrating .kibana-6_8 saved objects to .kibana_9"}
Sep 09 11:33:30 kibana0 kibana[2133]: {"type":"log","@timestamp":"2020-09-09T11:33:30Z","tags":["fatal","root"],"pid":2133,"message":"{ Error: Document contains at least one immense term in field=\"upgrade-assistant-reindex-operation.errorMessage\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[91, 109, 97, 112, 112, 101, 114, 95, 112, 97, 114, 115, 105, 110, 103, 95, 101, 120, 99, 101, 112, 116, 105, 111, 110, 93, 32, 91, 105, 110]...', original message: bytes can be at most 32766 in length; got 34044\n at Object.write (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:152:21)\n at process._tickCallback (internal/process/next_tick.js:68:7)\n detail:\n { index:\n { _index: '.kibana_9',\n _type: '_doc',\n _id:\n 'upgrade-assistant-reindex-operation:3d5e8680-ffd6-11e9-9023-e5862c2f29f3',\n status: 400,\n error: [Object] } } }"}To complete the upgrade, I did the following:
./node_modules/.bin/elasticdump --type=data --input http://esnode1.internal.softwareheritage.org:9200/.kibana-6 --output kibana-6.json
./node_modules/.bin/elasticdump --type=mapping --input http://esnode1.internal.softwareheritage.org:9200/.kibana-6 --output http://esnode1.internal.softwareheritage.org:9200/.kibana_1 ./node_modules/.bin/elasticdump --type=data --input kibana-6.json --output http://esnode1.internal.softwareheritage.org:9200/.kibana_1
When restarting kibana again, the migration could complete and it looks like the existing dashboards have been restored.