Upgrade kafka nodes to bullseye
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	vsellier
	Dec 2 2021, 10:13 AM

Description

Upgrade the kafa nodes:

~~journal0.internal.staging.swh.network~~ (staging's kafka was migrated to storage1, which can't be upgraded to bullseye as swh-storage and swh-objstorage are deployed on it)
kafka[1-4].internal.softwareheritage.org

Upgrade plan:

disable puppet
upgrade buster packages
stop and disable kafka
stop and disable zookeeper
reboot if needed
upgrade to bullseye following the plan described in T3749
Try to reboot, if the network is not working
- edit the /etc/network/interfaces file and comment the physical interface declarations

auto enp94s0f0np0
iface enp94s0f0np0 inet manual
  mtu 9000
# interface enp94s0f1np1 configuration
auto enp94s0f1np1
iface enp94s0f1np1 inet manual
  mtu 9000

- Remove the interface configuration on puppet (kafka[1-4].internal.softwareheritage.org` files) before restarting puppet
restart and check zookeeper status
restart and check kafka status (wait for the cluster to recover[1])
re-enable puppet

[1] http://getty.internal.softwareheritage.org:9000/clusters/rocquencourt/topics

Revisions and Commits

rSPSITE puppet-swh-site
	D6736	rSPSITEca930fb95203 kafka: remove the physical interface declaration
	D6733	rSPSITE40bbfa808515 kafka: remove the physical interface declaration
	D6732	rSPSITEd8bd43fcacc6 kafka: remove the physical interface declaration
	D6728	rSPSITEf752bfd2658d debian: prepare the kafka servers upgrade

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T3579 Meta-task: upgrade infrastructure to Debian Bullseye
		Migrated	gitlab-migration	T3762 Upgrade kafka nodes to bullseye

Event Timeline

vsellier changed the task status from Open to Work in Progress.Dec 2 2021, 10:13 AM

vsellier triaged this task as Normal priority.

vsellier created this task.

vsellier updated the task description. (Show Details)Dec 2 2021, 10:37 AM

vsellier added a revision: D6728: debian: prepare the kafka servers upgrade.Dec 2 2021, 2:49 PM

vsellier added a commit: rSPSITEf752bfd2658d: debian: prepare the kafka servers upgrade.Dec 2 2021, 3:04 PM

vsellier updated the task description. (Show Details)Dec 2 2021, 3:38 PM

vsellier updated the task description. (Show Details)Dec 2 2021, 3:49 PM

Actual actions

As root user on kafka1:

# disable puppet
puppet agent --disable 'T3762'
# upgrade buster packages
apt update
apt upgrade -y
# stop kafka
systemctl stop kafka.service; systemctl disable kafka.service
# stop zookeeper
systemctl stop zookeeper.service; systemctl disable zookeeper.service
# status before reboot
uname -a
Linux kafka1 5.10.0-0.bpo.3-amd64 #1 SMP Debian 5.10.13-1~bpo10+1 (2021-02-11) x86_64 GNU/Linux
uptime
14:56:00 up 258 days, 21:33,  5 users,  load average: 1.05, 7.36, 7.38

reboot

After reboot, let's do the actual upgrade:

sed -i -e 's/buster/bullseye/' /etc/apt/sources.list.d/*
sed -i -e 's,bullseye/updates,bullseye-security,' /etc/apt/sources.list.d/debian-security.list
apt update
apt upgrade -y
apt dist-upgrade -y
uname -a
Linux kafka1 5.10.0-0.bpo.9-amd64 #1 SMP Debian 5.10.70-1~bpo10+1 (2021-10-10) x86_64 GNU/Linux
uptime
 15:10:31 up 12 min,  5 users,  load average: 4.50, 2.28, 1.06
reboot

As kafka machines are actual hardware, connected through the idrac just in case
something goes wrong during reboot.

The network interfaces are not up at reboot.
We need to adapt them a bit.

vsellier added a revision: D6732: kafka: remove the physical interface declaration.Dec 2 2021, 4:39 PM

vsellier added a commit: rSPSITEd8bd43fcacc6: kafka: remove the physical interface declaration.Dec 2 2021, 4:43 PM

After some adaptation in the network interface ^, the network is back at reboot.
Activated back zookeeper and kafka and everything is ok for kafka1.
Inventory updated for that node from buster to bullseye.

*next step*: migrate remaining kafka[2-4]

vsellier added a revision: D6733: kafka: remove the physical interface declaration.Dec 2 2021, 5:16 PM

vsellier added a commit: rSPSITE40bbfa808515: kafka: remove the physical interface declaration.Dec 2 2021, 5:22 PM

kafka2 migrated as well, services up, inventory updated.

*next step*: migrate remaining kafka[3-4]

vsellier added a revision: D6736: kafka: remove the physical interface declaration.Dec 2 2021, 5:38 PM

vsellier added a commit: rSPSITEca930fb95203: kafka: remove the physical interface declaration.Dec 2 2021, 5:40 PM

kafka3 migrated as well, services up, inventory updated.

*next step*: migrate remaining kafka4

kafka4 migrated as well, services up, inventory updated.

We are done ;)

ardumont closed this task as Resolved.Dec 2 2021, 6:22 PM