Page MenuHomeSoftware Heritage

Make logstash on banco store documents on Elasticsearch version 6.x nodes
Closed, ResolvedPublic

Description

Starting from version 6.0, Elasticsearch no longer creates indices with more than one mapping type.
In order to create new indices with a forward-compatible format, the Logstash instance on Banco must thus use an Elasticsearch 6.x node as output target.

Event Timeline

ftigeot created this task.Mar 6 2018, 3:35 PM
ftigeot triaged this task as Normal priority.

Reference documentation wrt document type removal in Elasticsearch:
https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html

ftigeot renamed this task from Upgrade logstash on banco to version 6.x to Make logstash on banco store documents on Elasticsearch version 6.x nodes.May 23 2018, 3:50 PM
ftigeot updated the task description. (Show Details)
ftigeot added a comment.EditedMay 23 2018, 4:03 PM

Elasticsearch 6.x is also unable to write new data to indexes created with more than one mapping type (the default on previous versions).

ftigeot closed this task as Resolved.May 24 2018, 12:47 PM

Logstash configuration on banco changed to inject data on the esnode1 and 2 Elasticsearch instances:

diff --git a/logstash/conf.d/output.conf b/logstash/conf.d/output.conf
index ffa134a..2d6e703 100644
--- a/logstash/conf.d/output.conf
+++ b/logstash/conf.d/output.conf
@@ -1,12 +1,18 @@
 output {
     if "swh-worker@" in [systemd_unit] {
         elasticsearch {
-               hosts => ["localhost:9200"]
+               hosts => [
+                       "esnode1.internal.softwareheritage.org:9200",
+                       "esnode2.internal.softwareheritage.org:9200"
+               ]
                 index => "swh_workers-%{+YYYY.MM.dd}"
         }
     } else {
         elasticsearch {
-               hosts => ["localhost:9200"]
+               hosts => [
+                       "esnode1.internal.softwareheritage.org:9200",
+                       "esnode2.internal.softwareheritage.org:9200"
+               ]
                 index => "systemlogs-%{+YYYY.MM.dd}"
         }
     }

No visible issues so far.