Page MenuHomeSoftware Heritage
Create Task
Maniphest T4320

Upgrade gitlab instance to v15
Closed, MigratedEdits Locked
Actions

Description

Prior to reset the state and run the migration again, we'd like to check the migration
process with a gitlab instance with some more data than the previous migration.

Plan:

  • Determine operator version: 0.6.3 [2]
  • Determine gitlab chart version: 5.9.3 (gitlab 14.9.3) [3]
  • noop: Upgrade to the latest available chart version: 5.9.3 (for 0.6.3)
  • Identify the desired Operator version: 0.8.0 [4]
  • T4320#86633: Backup gitlab to revert if problem [5]
  • T4320#86671: Upgrade gitlab operator [1] to 0.8.0
  • T4320#86671: Then upgrade gitlab chart version successively from 5.94 -> 6.0.0 [6]

[1] https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/operator_upgrades.md

[2] https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w?view#Deploy-the-gitlab-operator

[3]

$ kubectl -n gitlab-system logs deployment/gitlab-controller-manager -c manager | grep -i version | tail -1
2022-05-31T14:32:02.401Z        DEBUG   controllers.GitLab      version information     {"gitlab": "gitlab-system/gitlab", "upgrade": false, "current version": "5.9.3", "desired version": "5.9.3"}

[4] https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/0.8.0/CHART_VERSIONS

[5] https://docs.gitlab.com/charts/backup-restore/backup.html

[6] https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/gitlab_upgrades.md

Related Objects
Search...

Event Timeline

ardumont triaged this task as High priority.Jun 9 2022, 4:13 PM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont mentioned this in T4064: Test GitLab migration scripts.Jun 9 2022, 5:19 PM
ardumont updated the task description. (Show Details)Jun 9 2022, 5:51 PM
ardumont updated the task description. (Show Details)Jun 9 2022, 6:04 PM
ardumont updated the task description. (Show Details)Jun 9 2022, 6:08 PM
ardumont updated the task description. (Show Details)Jun 10 2022, 9:55 AM
ardumont changed the task status from Open to Work in Progress.Jun 10 2022, 10:06 AM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont added a comment.Jun 10 2022, 11:08 AM

Backup done:

$ kubectl exec gitlab-toolbox-867b6c8f49-hx9rz -it -- backup-utility
Defaulted container "toolbox" out of: toolbox, certificates (init), configure (init)
2022-06-10 07:49:17 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-06-10 07:49:31 +0000 -- done
2022-06-10 07:49:43 +0000 -- Dumping repositories ...
...
[DONE] Backup can be found at s3://gitlab-backups/1654847341_2022_06_10_14.9.3_gitlab_backup.tar

Accessible from [1] with access credentials [2]

[1] https://minio-staging.swh.network/minio/gitlab-backups/

[2] https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w#Prerequisites

ardumont updated the task description. (Show Details)Jun 10 2022, 2:02 PM
ardumont updated the task description. (Show Details)Jun 10 2022, 4:50 PM
ardumont added a comment.Jun 10 2022, 4:53 PM
  • Upgrade gitlab operator
VERSION=0.8.0
kubectl apply -f \
  https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml

And make sure the new manager is happy

$ kubectl get pods  | grep manager
gitlab-controller-manager-788d4dbb78-46m86             2/2     Running       0          26s
gitlab-controller-manager-c4dcbbb96-pds8t              0/2     Terminating   3          21d
$ kubectl get pods | grep manager
gitlab-controller-manager-788d4dbb78-46m86             2/2     Running     0          2m55s
$ kubectl logs -f gitlab-controller-manager-788d4dbb78-46m86 manager | grep ConfigError
2022-06-10T12:17:58.056Z        DEBUG   events  Warning {"object": {"kind":"GitLab","namespace":"gitlab-system","name":"gitlab","uid":"da614c45-a261-49a6-99a6-3632900bfd81","apiVersion":"apps.gitlab.com/v1beta1","resourceVersion":"34179434"}, "reason": "ConfigError", "message": "Configuration error detected: chart version 5.9.3 not supported; please use one of the following: 5.9.4, 6.0.0, 5.10.3"}
  • Then bump the chart to one of the compatible version (5.9.4) first.
  • After some fight with persistent volume, the got migration.
$ kubectl logs gitlab-controller-manager-788d4dbb78-46m86 manager | grep version | tail -1
2022-06-10T13:47:55.040Z        DEBUG   controllers.GitLab      version information     {"gitlab": "gitlab-system/gitlab", "upgrade": false, "current version": "5.9.4", "desired version": "5.9.4"}
$ kubectl get gitlabs -n gitlab-system
NAME     STATUS    VERSION
gitlab   Running   5.9.4
  • Then bump to 0.6.0
$ kubectl get gitlabs -n gitlab-system
NAME     STATUS    VERSION
gitlab   Running   6.0.0

Note: We had to struggle a bit with pods that fail to start. So the deployment gets
stuck, it's unstucking by destroying deployments and then apply again the custom gitlab
deployment file [1] again.

[1]

$ kubectl apply -f gitlab-staging.yaml
ardumont updated the task description. (Show Details)Jun 10 2022, 4:53 PM
olasd added a subscriber: olasd.Jun 10 2022, 6:41 PM

After killing the minio data store, we've decided (as we had planned anyway) to start again from a clean slate.

Removal of resources:

# delete gitlab instance
kubectl delete gitlab gitlab

# delete gitlab operator (! will delete cert-manager custom resource definitions too...)
VERSION=0.8.0
kubectl delete -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"

# delete persistent volume connections
kubectl get pvc | grep 'gitlab'
kubectl get pvc | grep 'gitlab' | awk '{print $1}' | xargs kubectl delete pvc

# delete persistent volumes if any remain, e.g. with ReclaimPolicy=Remain (and you want to delete them)
kubectl get pv
kubectl get pv | grep 'gitlab' | awk '{print $1}' | xargs kubectl delete pv

# dump, review and delete autogenerated secrets
kubectl get -l app.kubernetes.io/managed-by=gitlab-operator -o yaml secrets > current-secrets.yaml
kubectl delete -l app.kubernetes.io/managed-by=gitlab-operator secrets --dry-run=client
kubectl delete -l app.kubernetes.io/managed-by=gitlab-operator secrets

# Uninstall cert-manager
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

Reinstall

# Install cert-manager (at least the CRDs are needed for the GitLab Operator chart, even if other resources seem to be defined again)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

# Install gitlab operator again
kubectl apply -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"

# Check that it's running
kubectl get pods | grep controller | grep -i running

# Redefine the gitlab resource
kubectl apply -f gitlab-staging.yaml

# Wait forever for it to come up
watch -d kubectl get pods

You can then login by recovering the initial root password from the newly regenerated secret:

kubectl get secret -n gitlab-system gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo

To avoid a new unscheduled data disassembly, we've reset all the physicalvolumes to have spec.persistentVolumeReclaimPolicy=Retain:

$ kubectl edit pv

# do the replacement and save

persistentvolume/pvc-06a1379e-7bf6-446b-bbea-2b7a438837c9 edited
persistentvolume/pvc-42c5110b-9462-4367-87d1-925a5221d5df edited
persistentvolume/pvc-631a90af-d603-41ce-8874-21f78069a7bc edited
persistentvolume/pvc-7ed641c8-aef2-4a1a-8355-43d8e19ec8fe edited
persistentvolume/pvc-83e8d340-9a83-4ae7-90aa-e655eb8c5609 edited
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Jun 13 2022, 1:11 PM
ardumont closed this task as Resolved.Jun 22 2022, 12:52 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:07 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal