Page MenuHomeSoftware Heritage
Create Task
Maniphest T1273

elasticsearch: about the elk stack policy upgrade?
Closed, MigratedEdits Locked
Actions

Description

We got caught off-guard by a logstash upgrade (version was not completely pinned in our puppet manifest).

This is now fixed (D544).
We kept the latest version (6.4.2) as this fixed some issues on the indexed logs.

In general though, that begs the elasticsearch upgrade policy question.
Do we synchronize or not those version number between the elk stack (recently they started sharing the same numbering)?

This needs further digging to know if it's worth it or not to upgrade.
It's not a simple matter as the elasticsearch upgrade is apparently not easily done (manual intervention needed).

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Oct 17 2018, 3:36 PM
ardumont created this task.
ftigeot added a subscriber: ftigeot.Oct 17 2018, 3:49 PM

Upgrading the Elasticsearch cluster is a somewhat delicate operation since nodes running old Elasticsearch versions can no longer store new data but it is not really difficult to handle properly.
The biggest issue could be with Kibana / Elasticsearch interactions: some old Kibana versions are known to stop displaying dashboards when talking to newer Elasticsearch servers.

Upgrading just one Elasticsearch node and waiting for the cluster to become green again usually takes ~= 20 minutes; we could easily experiment on one node and roll back if necessary.

olasd added a project: System administration.Oct 17 2018, 5:06 PM
ftigeot added a comment.Oct 18 2018, 11:40 AM

A quick analysis of the 6.4.x family versions show they bring significant bug fixes to the table.
One particularly interesting aspect is general cluster reliability improvements when nodes leaves or come back to the cluster.

ftigeot added a comment.Oct 18 2018, 12:18 PM

Elasticsearch, Logstash and Kibana are now released together and similar versions are sure to be compatible. It makes sense to have a global Puppet constant defining which general elk stack version to use for packages.

This doesn't preclude an incremental deployment policy: we just don't need to run the Puppet agent on all hosts at the same time.

ardumont added a comment.Oct 18 2018, 3:18 PM

Related D548

olasd closed this task as Resolved.Jan 6 2020, 4:30 PM
olasd claimed this task.
olasd added a subscriber: olasd.

The puppet manifests have been updated to hardcode the ELK stack version, and ensure they stay there with package pinning.

Rolling upgrades can be done by overriding for specific hosts, until the full upgrade is over.

gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:54 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal