The machines have 4x 2TB HDDs
Partitions:
- RAID 1	/			40GB
- RAID 0	/srv/elasticsearch	all the rest

Using a RAID 0 volume will improve I/O performance
We don't need redundancy on /srv/elasticsearch since
Elasticsearch itself will provide it

Name the hosts with easy to remember names in the DNS like "esnode0.internal.softwareheritage.org"

[This stage could be puppetized]

The new hosts have 32GB of RAM.

Change etc/elasticsearch/jvmoptions :
	# Use a maximum of 50% of available memory for the jvm heap
	# Keep it <32GB in order to benefit from jvm pointer compression
	# Around 50% of memory should be kept free for disk caching purposes
	-Xms15g
	-Xmx15g

	# The default openjdk 1.8 garbage collector allows itself to stop all
	# operations for indefinite amounts of time, causing cluster issues after
	# a while.
	# Use G1GC instead, it has more realtime performance characteristics and
	# will help to keep the cluster green and responsive
	-XX:+UseG1GC 

	# Comment out or remove -XX CMS options

Change etc/elasticsearch/elasticsearch.yml

	# Store indexes in /srv/elasticsearch
	path.data:	/srv/elasticsearch

	# Since the es cluster is mostly used as an archive and few
	# search requests are being made daily, increase available
	# memory for indexing. (default is 10% of heap size)
	indices.memory.index_buffer_size: 50%

We should be using a dedicated hardware node but budget is tight
and performance requirements low.
Having logstash running on one of the Elasticsearch nodes shouldn't
cause many issues with the current workload as of 2018-03.

Logstash 6.x is the new version
We need to use it for new indexes in order to avoid compatibility
issues with Logstash/Journalbeat 5.x and Elasticsearch 6.x

Configure it to send data to the Elasticsearch instance on Banco

Change etc/elasticsearch.yml

cluster.name:	swh-logging-prod
node.name:	esnode${node}
network.host:	192.168.100.${node_last_ip_digit}
discovery.zen.ping.unicast.hosts: [
	"banco.internal.softwareheritage.org",
	"esnode0.internal.softwareheritage.org",
	"esnode1.internal.softwareheritage.org",
	"esnode2.internal.softwareheritage.org"
]

# We have three nodes, avoid split-brains
discovery.zen.minimum_master_nodes: 2

This parameter is extremely important. Without it, we can get split-brains.

Wait for the cluster to turn green

# Limit the maximum amount of I/Os dedicated to Lucene segment mergings
# (because we use slow HDDs)
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
	"index.merge.scheduler.max_thread_count" : "1"
}'

Some load-balancer / error detection system could be useful if the cluster
gets big.
For three nodes, hardcoding all of them should be fine.

Logstash outputs accepts a list of hosts, this should be enough for
a small cluster.