Retrieve and test the migration scripts
Follow up in [1]
[1] https://hedgedoc.softwareheritage.org/ezUtxkTDSJ-JBLARYrWIjA?view
Description
Description
Revisions and Commits
Revisions and Commits
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Open | None | T2221 Development workflow & code quality | ||
| Open | olasd | T2225 Migrate to GitLab | ||
| Work in Progress | None | T4064 Test GitLab migration scripts |
Event Timeline
Comment Actions
Gist of the actions are currently:
- Mirror the forgerie repository to add some docker commands to allow sandboxed execution (see diffs ^)
- Adaptations in the forgerie code source to allow migration runs with our current gitlab/phabricator instances (see diffs ^)
- Current runs are working a bit but do not succeed entirely
More information to come during the day.
Comment Actions
Clean up gitlab instance
$ kubectx euwest-gitlab-staging $ kubens gitlab-system # delete gitlab instance $ kubectl delete -f gitlab-staging.yaml gitlab.apps.gitlab.com "gitlab" deleted # Delete operator $ VERSION=0.6.3 $ kubectl delete -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml" # once the pods are no longer running # delete pvc once the gitlab cluster is $ kubectl get pvc | grep 'gitlab' | awk '{print $1}' | xargs -t -r kubectl delete pvc kubectl delete pvc data-gitlab-postgresql-0 export-gitlab-minio-0 redis-data-gitlab-redis-master-0 repo-data-gitlab-gitaly-0 persistentvolumeclaim "data-gitlab-postgresql-0" deleted persistentvolumeclaim "export-gitlab-minio-0" deleted persistentvolumeclaim "redis-data-gitlab-redis-master-0" deleted persistentvolumeclaim "repo-data-gitlab-gitaly-0" deleted
Start fresh gitlab instance
$ VERSION=0.6.3 $ kubectl apply -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml" $ kubectl get pods | grep controller | grep -i running # once controller is running $ kubectl apply -f gitlab-staging.yaml gitlab.apps.gitlab.com/gitlab created # watch gitlab being installed $ watch kubectl get pods $ while true; do kubectl -n gitlab-system logs deployment/gitlab-controller-manager -c manager -f ; sleep 2; done
Configure access token
- Retrieve the root credentials
$ kubectl get secret -n gitlab-system gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo- Connect to the root user through web view
- User > Preferences > Access Tokens > Create access token (check all permissions)
Run migration
Prepare configuration
- clone https://forge.softwareheritage.org/source/forgerie/
- Adapt config/config.lisp:
(setf forgerie-core:*working-directory* "/srv/forgerie/")
; insert your public key (and make sure the key has no passphrase and the container has
; access to it)
(setf forgerie-gitlab:*ssh-public-key* "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG7F8I7695YzFsZlBVk2145dHui04ZVxfbZb6G1yKepX tony@yavin4")
(setf forgerie-gitlab:*private-token* "<private-token>")
(setf forgerie-gitlab:*server-address* "https://gitlab-staging.swh.network/")
(setf forgerie-gitlab:*rails-command* "/usr/bin/kubectl")
(setf forgerie-gitlab:*rails-command-args* '("exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console"))
(setf forgerie-phabricator:*database-host* "127.0.0.1")
(setf forgerie-phabricator:*database-username* "root")
(setf forgerie-phabricator:*database-password* "<redacted>")
(setf forgerie-phabricator:*storage-location* "/srv/phabricator/storage/")
...
(setf forgerie-phabricator:*included-repositories*
'("puppet-environment")); only 1 for now- Tunnel ssh over tate to expose phabricator's mysql port
ssh -L 3306:localhost:3306 tate
Run
$ cd forgerie $ docker run \ -v $SWH_ENVIRONMENT_HOME/../forgerie:/opt/forgerie/ \ -v /srv/phabricator:/srv/phabricator \ -v ~/.kube:/srv/forgerie/.kube \ -v /var/tmp/migrate-gitlab/forgerie:/tmp/forgerie \ -v ~/.ssh/:/srv/forgerie/.ssh \ --name forgerie --net=host -it forgerie /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"
Note: --net=host to expose host network interfaces to the container
postmortem
It breaks when trying to push on the 'gitlab' remote origin of the puppet-environment
repository [1].
The repository is cloned but the remote origin gitlab is not configured somehow.
forgerie@yavin4:/$ cd /tmp/forgerie/phabricator/puppet-environment/ forgerie@yavin4:/tmp/forgerie/phabricator/puppet-environment$ git remote -v origin /srv/phabricator/repos/SENV/ (fetch) origin /srv/phabricator/repos/SENV/ (push)
[1] https://forge.softwareheritage.org/source/forgerie/browse/main/src/main/gitlab/export.lisp$370