Starting from version 6.0, Elasticsearch no longer creates indices with more than one mapping type.
In order to create new indices with a forward-compatible format, the Logstash instance on Banco must thus use an Elasticsearch 6.x node as output target.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T792 Make the elasticsearch logging cluster actually a cluster | ||
| Migrated | gitlab-migration | T983 Make logstash on banco store documents on Elasticsearch version 6.x nodes |
Event Timeline
Comment Actions
Reference documentation wrt document type removal in Elasticsearch:
https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
Comment Actions
Elasticsearch 6.x is also unable to write new data to indexes created with more than one mapping type (the default on previous versions).
Comment Actions
Logstash configuration on banco changed to inject data on the esnode1 and 2 Elasticsearch instances:
diff --git a/logstash/conf.d/output.conf b/logstash/conf.d/output.conf index ffa134a..2d6e703 100644 --- a/logstash/conf.d/output.conf +++ b/logstash/conf.d/output.conf @@ -1,12 +1,18 @@ output { if "swh-worker@" in [systemd_unit] { elasticsearch { - hosts => ["localhost:9200"] + hosts => [ + "esnode1.internal.softwareheritage.org:9200", + "esnode2.internal.softwareheritage.org:9200" + ] index => "swh_workers-%{+YYYY.MM.dd}" } } else { elasticsearch { - hosts => ["localhost:9200"] + hosts => [ + "esnode1.internal.softwareheritage.org:9200", + "esnode2.internal.softwareheritage.org:9200" + ] index => "systemlogs-%{+YYYY.MM.dd}" } }
No visible issues so far.