Subtask dedicated to the rancher's kubernetes cluster upgrade
Description
Description
Status | Assigned | Task | ||
---|---|---|---|---|
Migrated | gitlab-migration | T4358 Upgrade AKS versions | ||
Migrated | gitlab-migration | T4359 Update rancher cluster to kubernetes 1.22 |
Event Timeline
Comment Actions
The kubernetes upgrade was launched through the azure portal (it's also possible to trigger it with the az command line)
Everything looks fine:
- A new node with the version 1.22.6 was triggerd
kubectl get pods -o wide; echo; kubectl get nodes -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES debian 1/1 Running 1 (23m ago) 27m 10.244.0.63 aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-5vlq6 1/1 Running 0 91m 10.244.0.59 aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-92txx 1/1 Running 0 90m 10.244.0.60 aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-cfshs 1/1 Running 0 91m 10.244.0.58 aks-default-36212332-vmss000000 <none> <none> rancher-webhook-6958cfcddf-2gjwn 1/1 Running 0 85d 10.244.0.26 aks-default-36212332-vmss000000 <none> <none> NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-default-36212332-vmss000000 Ready agent 85d v1.21.9 10.240.0.5 <none> Ubuntu 18.04.6 LTS 5.4.0-1073-azure containerd://1.4.12+azure-3 aks-default-36212332-vmss000001 NotReady agent 19s v1.22.6 10.240.0.7 <none> Ubuntu 18.04.6 LTS 5.4.0-1085-azure containerd://1.5.11+azure-2
- The old 1.21.9 node was drained and all the load moved to the new node
kubectl get pods -o wide; echo; kubectl get nodes -o wide 18:24:53 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rancher-59f4c74c6f-dxxh4 1/1 Running 0 3m22s 10.244.1.8 aks-default-36212332-vmss000001 <none> <none> rancher-59f4c74c6f-wdrrq 1/1 Running 0 3m22s 10.244.1.6 aks-default-36212332-vmss000001 <none> <none> rancher-59f4c74c6f-zhwdn 1/1 Running 0 3m21s 10.244.1.3 aks-default-36212332-vmss000001 <none> <none> rancher-webhook-6958cfcddf-fcp75 1/1 Running 0 3m20s 10.244.1.16 aks-default-36212332-vmss000001 <none> <none> NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-default-36212332-vmss000000 Ready,SchedulingDisabled agent 85d v1.21.9 10.240.0.5 <none> Ubuntu 18.04.6 LTS 5.4.0-1073-azure containerd://1.4.12+azure-3 aks-default-36212332-vmss000001 Ready agent 3m56s v1.22.6 10.240.0.7 <none> Ubuntu 18.04.6 LTS 5.4.0-1085-azure containerd://1.5.11+azure-2
- unexpected A new 1.22.6 node was recreate, I think to reset the node pool to 0 and the first one deleted
kubectl get pods -o wide; echo; kubectl get nodes -o wide 18:30:35 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rancher-59f4c74c6f-7rksq 0/1 ContainerCreating 0 41s <none> aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-ptxbh 0/1 ContainerCreating 0 40s <none> aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-xlz4j 0/1 ContainerCreating 0 39s <none> aks-default-36212332-vmss000000 <none> <none> rancher-webhook-6958cfcddf-cqz42 1/1 Running 0 40s 10.244.2.5 aks-default-36212332-vmss000000 <none> <none> NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-default-36212332-vmss000000 Ready agent 98s v1.22.6 10.240.0.5 <none> Ubuntu 18.04.6 LTS 5.4.0-1085-azure containerd://1.5.11+azure-2
At the end, everything is ok:
kubectl get pods -o wide; echo; kubectl get nodes -o wide 18:30:37 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES rancher-59f4c74c6f-7rksq 1/1 Running 0 7m32s 10.244.2.9 aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-ptxbh 1/1 Running 0 7m31s 10.244.2.6 aks-default-36212332-vmss000000 <none> <none> rancher-59f4c74c6f-xlz4j 1/1 Running 0 7m30s 10.244.2.11 aks-default-36212332-vmss000000 <none> <none> rancher-webhook-6958cfcddf-cqz42 1/1 Running 0 7m31s 10.244.2.5 aks-default-36212332-vmss000000 <none> <none> NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME aks-default-36212332-vmss000000 Ready agent 8m29s v1.22.6 10.240.0.5 <none> Ubuntu 18.04.6 LTS 5.4.0-1085-azure containerd://1.5.11+azure-2