Upgrade the ELK stack servers to bullseye:
- esnode[1-3]
- logstash0
- kibana0
Description
Description
Revisions and Commits
Revisions and Commits
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T3579 Meta-task: upgrade infrastructure to Debian Bullseye | ||
| Migrated | gitlab-migration | T3750 Upgrade ELK stack to bullseye |
Event Timeline
Comment Actions
logstash0 upgrade
- remove useless packages
root@logstash0:~# apt list --installed | grep jdk openjdk-11-jre-headless/oldstable,oldstable,now 11.0.12+7-2~deb10u1 amd64 [installed] openjdk-8-jre-headless/now 8u222-b10-1~deb9u1 amd64 [installed,local] root@logstash0:~# apt list --installed | grep jdk WARNING: apt does not have a stable CLI interface. Use with caution in scripts. openjdk-11-jre-headless/oldstable,oldstable,now 11.0.12+7-2~deb10u1 amd64 [installed] openjdk-8-jre-headless/now 8u222-b10-1~deb9u1 amd64 [installed,local] root@logstash0:~# apt remove openjdk-11-jre-headless openjdk-8-jre-headless Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: fontconfig-config fonts-dejavu-core java-common libasound2 libasound2-data libavahi-client3 libavahi-common-data libavahi-common3 libcups2 libfontconfig1 libgraphite2-3 libharfbuzz0b libjpeg62-turbo liblcms2-2 libnspr4 libnss3 libpcsclite1 libxi6 libxrender1 libxtst6 x11-common Use 'apt autoremove' to remove them. The following packages will be REMOVED: ca-certificates-java openjdk-11-jre-headless openjdk-8-jre-headless 0 upgraded, 0 newly installed, 3 to remove and 9 not upgraded. After this operation, 270 MB disk space will be freed. Do you want to continue? [Y/n] y ...
- prepare the upgrade
root@logstash0:~# puppet agent --disable root@logstash0:~# sed -i -e 's/buster/bullseye/' /etc/apt/sources.list.d/* root@logstash0:~# sed -i -e 's,bullseye/updates,bullseye-security,' /etc/apt/sources.list.d/debian-security.list root@logstash0:/etc/apt# apt update Get:1 http://deb.debian.org/debian bullseye-backports InRelease [43.7 kB] Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [44.1 kB] Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB] Get:4 http://deb.debian.org/debian bullseye InRelease [116 kB] Get:5 http://packages.icinga.com/debian icinga-bullseye InRelease [8459 B] Ign:6 https://debian.softwareheritage.org bullseye-swh InRelease Get:7 https://debian.softwareheritage.org bullseye-swh Release [2302 B] Ign:8 https://debian.softwareheritage.org bullseye-swh Release.gpg Get:9 https://debian.softwareheritage.org bullseye-swh/main amd64 Packages [2651 B] Hit:10 https://artifacts.elastic.co/packages/7.x/apt stable InRelease Get:11 http://deb.debian.org/debian bullseye-backports/main amd64 Packages [138 kB] Get:12 http://deb.debian.org/debian bullseye-backports/main Translation-en [102 kB] Get:13 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [96.2 kB] Get:14 http://deb.debian.org/debian-security bullseye-security/main Translation-en [59.7 kB] Get:15 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [2592 B] Get:16 http://deb.debian.org/debian bullseye-updates/main Translation-en [2343 B] Get:17 http://deb.debian.org/debian bullseye/main amd64 Packages [8180 kB] Get:18 http://packages.icinga.com/debian icinga-bullseye/main amd64 Packages [16.6 kB] Get:19 http://deb.debian.org/debian bullseye/main Translation-en [6241 kB] Fetched 15.1 MB in 3s (4376 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 481 packages can be upgraded. Run 'apt list --upgradable' to see them. root@logstash0:/etc/apt# apt upgrade -y ... root@logstash0:/etc/apt# apt dist-upgrade -y root@logstash0:~# reboot root@logstash0:~# puppet agent --enable; puppet agent --test ...
- Inventory updated [1]
Everything looks good
[1] https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/15/
Comment Actions
kibana0
- The server was updated using the same procedure used for logstash0
- there is no error detected when puppet is running
- all the services are correctly started
Comment Actions
esnode[1-3]
The upgrade by itself was made with the same command as explained previously.
The shard allocation was disabled during the process to avoid unnecessary movements of shard in the cluster
cat > /tmp/shard_allocation.json <<EOF
{
"persistent": {
"cluster.routing.allocation.enable": ["primaries"|null] <---- disabled | enabled
}
}
EOF
curl -H'content-type: application/json' -XPUT http://192.168.100.61:9200/_cluster/settings -d @/tmp/shard_allocation.jsonAfter the reboot, the bounding interface was down.
The configuration of the interfaces needed to be updated to removing the raw interfaces configuration
Source:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968368
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990428
for example for esnode2:
vsellier@esnode2 /etc/network % diff -U 3 interfaces.buster interfaces --- interfaces.buster 2021-11-25 16:52:51.420640148 +0000 +++ interfaces 2021-11-25 18:19:35.782819294 +0000 @@ -16,11 +16,11 @@ # dns-nameservers 192.168.100.29 # dns-search internal.softwareheritage.org -auto enp2s0f0 -iface enp2s0f0 inet manual +#auto enp2s0f0 +#iface enp2s0f0 inet manual -auto enp2s0f1 -iface enp2s0f1 inet manual +#auto enp2s0f1 +#iface enp2s0f1 inet manual auto bond0 iface bond0 inet manual