Page MenuHomeSoftware Heritage

Upgrade ELK stack to bullseye
Closed, ResolvedPublic

Description

Upgrade the ELK stack servers to bullseye:

  • esnode[1-3]
  • logstash0
  • kibana0

Event Timeline

vsellier changed the task status from Open to Work in Progress.Thu, Nov 25, 9:53 AM
vsellier triaged this task as Normal priority.
vsellier created this task.

logstash0 upgrade

  • remove useless packages
root@logstash0:~# apt list --installed | grep jdk
openjdk-11-jre-headless/oldstable,oldstable,now 11.0.12+7-2~deb10u1 amd64 [installed]
openjdk-8-jre-headless/now 8u222-b10-1~deb9u1 amd64 [installed,local]

root@logstash0:~# apt list --installed | grep jdk

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

openjdk-11-jre-headless/oldstable,oldstable,now 11.0.12+7-2~deb10u1 amd64 [installed]
openjdk-8-jre-headless/now 8u222-b10-1~deb9u1 amd64 [installed,local]
root@logstash0:~# apt remove openjdk-11-jre-headless openjdk-8-jre-headless    
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fontconfig-config fonts-dejavu-core java-common libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libfontconfig1 libgraphite2-3 libharfbuzz0b libjpeg62-turbo liblcms2-2 libnspr4 libnss3 libpcsclite1 libxi6 libxrender1 libxtst6
  x11-common
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  ca-certificates-java openjdk-11-jre-headless openjdk-8-jre-headless
0 upgraded, 0 newly installed, 3 to remove and 9 not upgraded.
After this operation, 270 MB disk space will be freed.
Do you want to continue? [Y/n] y
...
  • prepare the upgrade
root@logstash0:~# puppet agent --disable
root@logstash0:~# sed -i -e 's/buster/bullseye/' /etc/apt/sources.list.d/*
root@logstash0:~# sed -i -e 's,bullseye/updates,bullseye-security,' /etc/apt/sources.list.d/debian-security.list
root@logstash0:/etc/apt# apt update
Get:1 http://deb.debian.org/debian bullseye-backports InRelease [43.7 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [44.1 kB]                                                                                   
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]                                                                
Get:4 http://deb.debian.org/debian bullseye InRelease [116 kB]                                                                                
Get:5 http://packages.icinga.com/debian icinga-bullseye InRelease [8459 B]                                                                          
Ign:6 https://debian.softwareheritage.org bullseye-swh InRelease                                                        
Get:7 https://debian.softwareheritage.org bullseye-swh Release [2302 B]
Ign:8 https://debian.softwareheritage.org bullseye-swh Release.gpg    
Get:9 https://debian.softwareheritage.org bullseye-swh/main amd64 Packages [2651 B]
Hit:10 https://artifacts.elastic.co/packages/7.x/apt stable InRelease                   
Get:11 http://deb.debian.org/debian bullseye-backports/main amd64 Packages [138 kB]
Get:12 http://deb.debian.org/debian bullseye-backports/main Translation-en [102 kB]
Get:13 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [96.2 kB]
Get:14 http://deb.debian.org/debian-security bullseye-security/main Translation-en [59.7 kB]
Get:15 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2592 B]
Get:16 http://deb.debian.org/debian bullseye-updates/main Translation-en [2343 B]
Get:17 http://deb.debian.org/debian bullseye/main amd64 Packages [8180 kB]
Get:18 http://packages.icinga.com/debian icinga-bullseye/main amd64 Packages [16.6 kB]
Get:19 http://deb.debian.org/debian bullseye/main Translation-en [6241 kB]
Fetched 15.1 MB in 3s (4376 kB/s)                                                                  
Reading package lists... Done
Building dependency tree       
Reading state information... Done
481 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@logstash0:/etc/apt# apt upgrade -y
...
root@logstash0:/etc/apt# apt dist-upgrade -y
root@logstash0:~# reboot
root@logstash0:~# puppet agent --enable; puppet agent --test
...
  • Inventory updated [1]

Everything looks good

[1] https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/15/

kibana0

  • The server was updated using the same procedure used for logstash0
  • there is no error detected when puppet is running
  • all the services are correctly started

esnode[1-3]

The upgrade by itself was made with the same command as explained previously.
The shard allocation was disabled during the process to avoid unnecessary movements of shard in the cluster

cat > /tmp/shard_allocation.json <<EOF
{
  "persistent": {
    "cluster.routing.allocation.enable": ["primaries"|null]   <---- disabled | enabled
  }
}
EOF

curl -H'content-type: application/json' -XPUT http://192.168.100.61:9200/_cluster/settings -d @/tmp/shard_allocation.json

After the reboot, the bounding interface was down.
The configuration of the interfaces needed to be updated to removing the raw interfaces configuration
Source:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968368
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990428

for example for esnode2:

vsellier@esnode2 /etc/network % diff -U 3 interfaces.buster interfaces
--- interfaces.buster	2021-11-25 16:52:51.420640148 +0000
+++ interfaces	2021-11-25 18:19:35.782819294 +0000
@@ -16,11 +16,11 @@
 #	dns-nameservers 192.168.100.29
 #	dns-search internal.softwareheritage.org
 
-auto enp2s0f0
-iface enp2s0f0 inet manual
+#auto enp2s0f0
+#iface enp2s0f0 inet manual
 
-auto enp2s0f1
-iface enp2s0f1 inet manual
+#auto enp2s0f1
+#iface enp2s0f1 inet manual
 
 auto bond0
 iface bond0 inet manual

All the nodes are updated