Page MenuHomeSoftware Heritage

Test GitLab migration scripts
Started, Work in Progress, HighPublic

Description

Retrieve and test the migration scripts

Follow up in [1]

[1] https://hedgedoc.softwareheritage.org/ezUtxkTDSJ-JBLARYrWIjA?view

Related Objects

StatusAssignedTask
OpenNone
Openolasd
Work in ProgressNone

Event Timeline

bchauvet renamed this task from Gitlab migration tests to Test GitLab migration scripts.Mar 21 2022, 10:45 AM
bchauvet triaged this task as High priority.
bchauvet created this task.
ardumont changed the task status from Open to Work in Progress.Thu, May 19, 4:55 PM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont added a subscriber: ardumont.

Gist of the actions are currently:

  • Mirror the forgerie repository to add some docker commands to allow sandboxed execution (see diffs ^)
  • Adaptations in the forgerie code source to allow migration runs with our current gitlab/phabricator instances (see diffs ^)
  • Current runs are working a bit but do not succeed entirely

More information to come during the day.

Clean up gitlab instance

$ kubectx euwest-gitlab-staging
$ kubens gitlab-system
# delete gitlab instance
$ kubectl delete -f gitlab-staging.yaml
gitlab.apps.gitlab.com "gitlab" deleted
# Delete operator
$ VERSION=0.6.3
$ kubectl delete -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"
# once the pods are no longer running
# delete pvc once the gitlab cluster is
$ kubectl get pvc | grep 'gitlab' | awk '{print $1}' | xargs -t -r kubectl delete pvc
kubectl delete pvc data-gitlab-postgresql-0 export-gitlab-minio-0 redis-data-gitlab-redis-master-0 repo-data-gitlab-gitaly-0
persistentvolumeclaim "data-gitlab-postgresql-0" deleted
persistentvolumeclaim "export-gitlab-minio-0" deleted
persistentvolumeclaim "redis-data-gitlab-redis-master-0" deleted
persistentvolumeclaim "repo-data-gitlab-gitaly-0" deleted

Start fresh gitlab instance

$ VERSION=0.6.3
$ kubectl apply -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"
$ kubectl get pods | grep controller | grep -i running
# once controller is running
$ kubectl apply -f gitlab-staging.yaml
gitlab.apps.gitlab.com/gitlab created
# watch gitlab being installed
$ watch kubectl get pods
$ while true; do kubectl -n gitlab-system logs deployment/gitlab-controller-manager -c manager -f ; sleep 2; done

Configure access token

  • Retrieve the root credentials
$ kubectl get secret -n gitlab-system gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
  • Connect to the root user through web view
  • User > Preferences > Access Tokens > Create access token (check all permissions)

Run migration

Prepare configuration

(setf forgerie-core:*working-directory* "/srv/forgerie/")

; insert your public key (and make sure the key has no passphrase and the container has
; access to it)
(setf forgerie-gitlab:*ssh-public-key* "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG7F8I7695YzFsZlBVk2145dHui04ZVxfbZb6G1yKepX tony@yavin4")

(setf forgerie-gitlab:*private-token* "<private-token>")
(setf forgerie-gitlab:*server-address* "https://gitlab-staging.swh.network/")

(setf forgerie-gitlab:*rails-command* "/usr/bin/kubectl")
(setf forgerie-gitlab:*rails-command-args* '("exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console"))

(setf forgerie-phabricator:*database-host* "127.0.0.1")
(setf forgerie-phabricator:*database-username* "root")
(setf forgerie-phabricator:*database-password* "<redacted>")
(setf forgerie-phabricator:*storage-location* "/srv/phabricator/storage/")
...
(setf forgerie-phabricator:*included-repositories*
 '("puppet-environment")); only 1 for now
  • Tunnel ssh over tate to expose phabricator's mysql port
ssh -L 3306:localhost:3306 tate

Run

$ cd forgerie
$ docker run \
    -v $SWH_ENVIRONMENT_HOME/../forgerie:/opt/forgerie/ \
    -v /srv/phabricator:/srv/phabricator \
    -v ~/.kube:/srv/forgerie/.kube \
    -v /var/tmp/migrate-gitlab/forgerie:/tmp/forgerie \
    -v ~/.ssh/:/srv/forgerie/.ssh \
    --name forgerie --net=host -it forgerie
    /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"

Note: --net=host to expose host network interfaces to the container

postmortem

It breaks when trying to push on the 'gitlab' remote origin of the puppet-environment
repository [1].

The repository is cloned but the remote origin gitlab is not configured somehow.

forgerie@yavin4:/$ cd /tmp/forgerie/phabricator/puppet-environment/
forgerie@yavin4:/tmp/forgerie/phabricator/puppet-environment$ git remote -v
origin  /srv/phabricator/repos/SENV/ (fetch)
origin  /srv/phabricator/repos/SENV/ (push)

[1] https://forge.softwareheritage.org/source/forgerie/browse/main/src/main/gitlab/export.lisp$370