Page MenuHomeSoftware Heritage

Test GitLab migration scripts
Closed, MigratedEdits Locked

Description

Retrieve and test the migration scripts.

Plan:

  • Test migration
  • Ask for help by email if needed
  • Report status in [1]
  • Follow up to Frank Duncan and Karl Fogel by email
  • Waiting for feedback
  • T4064#86439: Trigger new migration run with fixes (crashes)
  • Status meeting
  • w/ olasd, ardumont: T4320: Upgrade gitlab
  • w/ olasd, ardumont: Clean up gitlab state
  • w/ olasd, ardumont: Trigger new migration run with fixes (running on bespin) for one repository
  • Analyse and troubleshoot and fix problems (latest one: T4064#86790)
  • Trigger back migration from scratch one repository
  • T4064#86791: Trigger back migration with another private repository
  • T4064#86792: Trigger with remaining sysadm repositories
  • Optimization (use http request pipelining)
  • T4064#87194: Successful run without human attendance
  • Analysis

[1] https://hedgedoc.softwareheritage.org/ezUtxkTDSJ-JBLARYrWIjA?view

Extra:

  • What to do with local patches to make the migration work? Propose upstream?

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Gist of the actions are currently:

  • Mirror the forgerie repository to add some docker commands to allow sandboxed execution (see diffs ^)
  • Adaptations in the forgerie code source to allow migration runs with our current gitlab/phabricator instances (see diffs ^)
  • Current runs are working a bit but do not succeed entirely

More information to come during the day.

Clean up gitlab instance

$ kubectx euwest-gitlab-staging
$ kubens gitlab-system
# delete gitlab instance
$ kubectl delete -f gitlab-staging.yaml
gitlab.apps.gitlab.com "gitlab" deleted
# Delete operator
$ VERSION=0.6.3
$ kubectl delete -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"
# once the pods are no longer running
# delete pvc once the gitlab cluster is
$ kubectl get pvc | grep 'gitlab' | awk '{print $1}' | xargs -t -r kubectl delete pvc
kubectl delete pvc data-gitlab-postgresql-0 export-gitlab-minio-0 redis-data-gitlab-redis-master-0 repo-data-gitlab-gitaly-0
persistentvolumeclaim "data-gitlab-postgresql-0" deleted
persistentvolumeclaim "export-gitlab-minio-0" deleted
persistentvolumeclaim "redis-data-gitlab-redis-master-0" deleted
persistentvolumeclaim "repo-data-gitlab-gitaly-0" deleted

Start fresh gitlab instance

$ VERSION=0.6.3
$ kubectl apply -f "https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${VERSION}/gitlab-operator-kubernetes-${VERSION}.yaml"
$ kubectl get pods | grep controller | grep -i running
# once controller is running
$ kubectl apply -f gitlab-staging.yaml
gitlab.apps.gitlab.com/gitlab created
# watch gitlab being installed
$ watch kubectl get pods
$ while true; do kubectl -n gitlab-system logs deployment/gitlab-controller-manager -c manager -f ; sleep 2; done

Configure access token

  • Retrieve the root credentials
$ kubectl get secret -n gitlab-system gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
  • Connect to the root user through web view
  • User > Preferences > Access Tokens > Create access token (check all permissions)

Run migration

Prepare configuration

(setf forgerie-core:*working-directory* "/srv/forgerie/")

; insert your public key (and make sure the key has no passphrase and the container has
; access to it)
(setf forgerie-gitlab:*ssh-public-key* "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG7F8I7695YzFsZlBVk2145dHui04ZVxfbZb6G1yKepX tony@yavin4")

(setf forgerie-gitlab:*private-token* "<private-token>")
(setf forgerie-gitlab:*server-address* "https://gitlab-staging.swh.network/")

(setf forgerie-gitlab:*rails-command* "/usr/bin/kubectl")
(setf forgerie-gitlab:*rails-command-args* '("exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console"))

(setf forgerie-phabricator:*database-host* "127.0.0.1")
(setf forgerie-phabricator:*database-username* "root")
(setf forgerie-phabricator:*database-password* "<redacted>")
(setf forgerie-phabricator:*storage-location* "/srv/phabricator/storage/")
...
(setf forgerie-phabricator:*included-repositories*
 '("puppet-environment")); only 1 for now
  • Tunnel ssh over tate to expose phabricator's mysql port
ssh -L 3306:localhost:3306 tate

Run

$ cd forgerie
$ docker run \
    -v $SWH_ENVIRONMENT_HOME/../forgerie:/opt/forgerie/ \
    -v /srv/phabricator:/srv/phabricator \
    -v ~/.kube:/srv/forgerie/.kube \
    -v /var/tmp/migrate-gitlab/forgerie:/tmp/forgerie \
    -v ~/.ssh/:/srv/forgerie/.ssh \
    --name forgerie --net=host -it forgerie
    /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"

Note: --net=host to expose host network interfaces to the container

postmortem

It breaks when trying to push on the 'gitlab' remote origin of the puppet-environment
repository [1].

The repository is cloned but the remote origin gitlab is not configured somehow.

forgerie@yavin4:/$ cd /tmp/forgerie/phabricator/puppet-environment/
forgerie@yavin4:/tmp/forgerie/phabricator/puppet-environment$ git remote -v
origin  /srv/phabricator/repos/SENV/ (fetch)
origin  /srv/phabricator/repos/SENV/ (push)

[1] https://forge.softwareheritage.org/source/forgerie/browse/main/src/main/gitlab/export.lisp$370

Summary sent to Frank Duncan and Karl Fogel.

postmortem

It breaks when trying to push on the 'gitlab' remote origin of the puppet-environment
repository [1].

The repository is cloned but the remote origin gitlab is not configured somehow.

forgerie@yavin4:/$ cd /tmp/forgerie/phabricator/puppet-environment/
forgerie@yavin4:/tmp/forgerie/phabricator/puppet-environment$ git remote -v
origin  /srv/phabricator/repos/SENV/ (fetch)
origin  /srv/phabricator/repos/SENV/ (push)

[1] https://forge.softwareheritage.org/source/forgerie/browse/main/src/main/gitlab/export.lisp$370

Turns out i was reading the wrong repository there ^. So my analysis is wrong. The
repository that interests us is in /tmp/forgerie/gitlab/puppet-environment. And this one
is properly configured.

What's actually missing is the ~/.ssh/config that goes with the ssh key configured in
the config.lisp read by the migration process.

I'm gonna attend to reply to the discussion and open a diff for that (for the
dockerfile).

To make the migration process restart, drop the repository ^ and drop the repository in
gitlab as well. Then trigger back the migration (currently running).

Migration is now ongoing without failing thus far.
Configured to only import puppet-environment for now [1] (to fix papercuts).
But that's still migrating the rest (users, issues, etc...).

[1] https://gitlab-staging.swh.network/software-heritage/puppet-environment

It seems to have finished badly [1].
But that may only be because it's configured to run in debug mode.
Latest run logs for later [2]

[2]

[1]

...
12: ((LAMBDA NIL :IN FORGERIE-CORE:RUN))
13: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
14: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-THREAD::RUN))
15: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
16: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::RUN))
17: (SB-THREAD::RUN)
18: ("foreign function: call_into_lisp")
19: ("foreign function: funcall1")

unhandled condition in --disable-debugger mode, quitting
Shutting down....
WARNING:
   Problem running exit hook #<FUNCTION (LAMBDA ()
                                          :IN
                                          FORGERIE-CORE:RUN) {1001A30F9B}>:
  In thread #<THREAD RUNNING {1001878103}>, attempt to join the current thread.
Shutting down....

Added 2 more repositories [1] to migrate and triggered it [2] [3]:

[1] config.lisp:

(setf forgerie-core:*debug* nil)
...
(setf forgerie-phabricator:*included-repositories*
 '("puppet-swh-site"
   "swh-sysadmin-provisioning"
   "puppet-environment"
))

[2]

$ time /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"
...

real    98m27.570s
user    36m34.774s
sys     49m26.570s
$ time /opt/forgerie/bin/postmortem | tee "/tmp/forgerie/postmortem-run-$(date +%Y%m%d-%H%M).log"
...

real    0m2.016s
user    0m0.792s
sys     0m0.153s

[3]

Another "private" (for the test) repository to try and migrate [1], it should be private but it's not [2]:

(setf forgerie-phabricator:*included-repositories*
 '(;; "swh-ansible"
   "training-mob-programming"  ;; private repository (well staff tagged)
   "puppet-swh-site"
   "swh-sysadmin-provisioning"
   "puppet-environment"
))

[1]

[2] https://gitlab-staging.swh.network/software-heritage/training-mob-programming

"Full" (with the overall sysadm repositories) run [1].

Results can be seen in the staging gitlab instance [2]

This migration run [2] imported the remaining repositories, it took:

real    414m4.028s
user    170m59.024s
sys     235m9.664s

[1]

[2] https://gitlab-staging.swh.network/

[3] The first import (see above) imported the initial issues. It's possible this run
also imported incrementally new issues.

fwiw, we're received notifications that the upstream repository have been delivered with some fixes.
So i've pulled the upstream branch and rebase the swh branch on it.

Then triggered a run. We'll see what happens.

well, no, computer says no ¯\_(ツ)_/¯

Unhandled FORGERIE-GITLAB::HTTP-ERROR in thread #<SB-THREAD:THREAD RUNNING
                                                   {1005560003}>:
  Http error code: 400, resp: (MESSAGE
                               (NAME (has already been taken) PATH
                                (has already been taken)))
Backtrace for: #<SB-THREAD:THREAD RUNNING {1005560003}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<FORGERIE-GITLAB::HTTP-ERROR {100430CEF3}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<FORGERIE-GITLAB::HTTP-ERROR {100430CEF3}>)
2: (INVOKE-DEBUGGER #<FORGERIE-GITLAB::HTTP-ERROR {100430CEF3}>)
3: (ERROR #<FORGERIE-GITLAB::HTTP-ERROR {100430CEF3}>)
4: (FORGERIE-GITLAB::MAKE-REQUEST "projects" :POST (("namespace_id" . "4") ("name" . "puppet-puppet-letsencrypt") ("path" . "puppet-puppet-letsencrypt") ("tag_list" . "language-puppet,system_administrators,sysadm,sync_to_github") ("issues_access_level" . #1="enabled") ("visibility" . "public") ("merge_requests_access_level" . #1#)) :SUDO NIL)
5: (FORGERIE-GITLAB::CREATE-PROJECT #S(FORGERIE-CORE::VC-REPOSITORY :NAME "puppet-puppet-letsencrypt" :SLUG "puppet-puppet-letsencrypt" :PRIMARY-PROJECTS NIL :GIT-LOCATION "/srv/phabricator/repos/192/" :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Language-Puppet" :TAGS ("language-puppet")) #S(FORGERIE-CORE::PROJECT :NAME "System administrators" :TAGS ("system_administrators" "sysadm")) #S(FORGERIE-CORE::PROJECT :NAME "Sync to GitHub" :TAGS ("sync_to_github"))) :PRIVATE NIL :COMMITS (#S(FORGERIE-CORE:COMMIT :SHA "61ffd8dee7c0f10f7ea7e84f23c4d732b7cd669b" :PARSED-COMMENT ("Merge pull request #265 from nod0n/unsupported_os
...
ardumont updated the task description. (Show Details)
ardumont changed the status of subtask T4320: Upgrade gitlab instance to v15 from Open to Work in Progress.Jun 10 2022, 10:06 AM

Failed [1] (full logs [2])

[1]

Backtrace for: #<SB-THREAD:THREAD RUNNING {1001890103}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {10034946C3}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {10034946C3}>)
2: (INVOKE-DEBUGGER #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {10034946C3}>)
3: (ERROR "Got error running git command ~A with args ~S in dir ~A" "push" ("gitlab" "--all") "/tmp/forgerie/gitlab/swh-sysadmin-provisioning")
4: (FORGERIE-CORE:GIT-CMD "/tmp/forgerie/gitlab/swh-sysadmin-provisioning" "push" ("gitlab" "--all") :ERROR T :DEBUG NIL :INPUT NIL)
5: (FORGERIE-GITLAB::CREATE-PROJECT #S(FORGERIE-CORE::VC-REPOSITORY :NAME "sysadm-provisioning" :SLUG "swh-sysadmin-provisioning" :PRIMARY-PROJECTS NIL :GIT-LOCATION "/srv/phabricator/repos/SPRE/" :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME
 "System administration" :TAGS ("system_administration")) #S(FORGERIE-CORE::PROJECT :NAME "System administrators" :TAGS ("system_administrators" "sysadm"))) :PRIVATE NIL :COMMITS (#S(FORGERIE-CORE:COMMIT :SHA "7b90b55041ccb14ebb15d8ed59c42
d2bb64ebfd3" :PARSED-COMMENT ("Align worker17 hardware spec to worker18
...
6: ((:METHOD FORGERIE-CORE:EXPORT-FORGE ((EQL :GITLAB) T)) #<unused argument> (:USERS (#S(FORGERIE-CORE::USER :USERNAME "administrator" :NAME "Software Heritage Administrator" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas+swhforgeadmi
n_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR NIL) #S(FORGERIE-CORE::USER :USERNAME "olasd" :NAME "Nicolas Dandrimont" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas.dandrimont_gmail.com@example.org" :IS-PRIMARY NIL) #S
(FORGERIE-CORE::EMAIL :ADDRESS "nicolas_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME #1="profile" :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbn
c463" :MIMETYPE #2="image/jpeg")) #S(FORGERIE-CORE::USER :USERNAME "ardumont" :NAME "Antoine R. Dumont" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "ardumont_softwareheritage.org@example.org" :IS-PRIMARY NIL) #S(FORGERIE-CORE::EMAIL :ADDRESS
 "antoine.romain.dumont_gmail.com@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 3 :NAME #3="profile" :SIZE 11009 :LOCATION "/tmp/forgerie/phabricator/files/wbp/PHID-FILE-nma6v42ey5ihn5i2uwbp" :MIMETYPE #4="image/
png")) #S(FORGERIE-CORE::USER :USERNAME "zack" :NAME "Stefano Zacchiroli" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5="profile" :SIZE 1
6531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6="image/jpeg")) #S(FORGERIE-CORE::USER :USERNAME "rdicosmo" :NAME "Roberto Di Cosmo" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "roberto_dicosmo
.org@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2875599 :NAME "profile" :SIZE 196883 :LOCATION "/tmp/forgerie/phabricator/files/uff/PHID-FILE-jck7nk5ue2ftdwq3cuff" :MIMETYPE "image/png")) #S(FORGERIE-CORE::USE
R :USERNAME "grouss" :NAME "Guillaume ROUSSEAU" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "guillaume_rousseaubottin.fr@example.org" :IS-PRIMARY NIL) #S(FORGERIE-CORE::EMAIL :ADDRESS "guillaume.rousseau_univ-paris-diderot.fr@example.org" :I
S-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FORGERIE-CORE::USER :USERNAME "swhphab" :NAME "SWH Phabricator Bot" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas+swhbot_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FO
RGERIE-CORE::USER :USERNAME "swh-sysadm-deploy" :NAME "Sysadmin Deploy User" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas+swh-sysadm-deploy_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FORGERIE-CORE::USER :US
ERNAME "swh-public-deploy" :NAME "Public Deploy User" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack+swhbot_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FORGERIE-CORE::USER :USERNAME "test-public" :NAME "Test User, w
ith no special permissions" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack+swhtest_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FORGERIE-CORE::USER :USERNAME "steff" :NAME "Laurent Steff" :EMAILS (#S(FORGERIE-CORE::E
MAIL :ADDRESS "laurent.steff_inria.fr@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) #S(FORGERIE-CORE::USER :USERNAME "dimitri" :NAME "Dimitri Fontaine" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "dimitri_2ndQuadrant.fr@example.org" :
IS-PRIMARY T)) :ADMIN NIL :AVATAR NIL) ...) :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Staff" :TAGS ("members" "staff")) #S(FORGERIE-CORE::PROJECT :NAME "Developers" :TAGS ("developers" "devel")) #S(FORGERIE-CORE::PROJECT :NAME "Language-
Python" :TAGS ("python" "language-python")) #S(FORGERIE-CORE::PROJECT :NAME "Language-Puppet" :TAGS ("language-puppet")) #S(FORGERIE-CORE::PROJECT :NAME "Origin-GitHub" :TAGS ("github" "gh" "origin-github")) #S(FORGERIE-CORE::PROJECT :NAME
 "Origin-CGit" :TAGS ("cgit" "origin-cgit")) #S(FORGERIE-CORE::PROJECT :NAME "System administrators" :TAGS ("system_administrators" "sysadm")) #S(FORGERIE-CORE::PROJECT :NAME "Format-Git" :TAGS ("git" "format-git")) #S(FORGERIE-CORE::PROJE
CT :NAME "Origin-Debian" :TAGS ("debian" "origin-debian")) #S(FORGERIE-CORE::PROJECT :NAME "Managers" :TAGS ("managers" "mgmt")) #S(FORGERIE-CORE::PROJECT :NAME "Paper authors" :TAGS ("paper_authors" "paper")) #S(FORGERIE-CORE::PROJECT :NA
ME "Format-Deb" :TAGS ("format-deb" "deb")) ...) :VC-REPOSITORIES (#S(FORGERIE-CORE::VC-REPOSITORY :NAME "sysadm-provisioning" :SLUG "swh-sysadmin-provisioning" :PRIMARY-PROJECTS NIL :GIT-LOCATION "/srv/phabricator/repos/SPRE/" :PROJECTS (
#S(FORGERIE-CORE::PROJECT :NAME #7="System administration" :TAGS #) #S(FORGERIE-CORE::PROJECT :NAME "System administrators" :TAGS #)) :PRIVATE NIL :COMMITS (#S(FORGERIE-CORE:COMMIT :SHA "7b90b55041ccb14ebb15d8ed59c42d2bb64ebfd3" :PARSED-CO
MMENT #) #S(FORGERIE-CORE:COMMIT :SHA "d7d5ff6b4b64cf76ed8738fd171edeec54abc4e8" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "6e7b13218cbcdbd7b8fd02f3e327f6924511ef67" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "179087d6fee28ad2b7
a47ff50ef02720bc707250" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "9a4d7636d7958b1c2793f06ebbbc0bf982c53b1d" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "51f802c63f79b64e2f6d294f1b53f9bdd08de6c6" :PARSED-COMMENT #) #S(FORGERIE-CO
RE:COMMIT :SHA "d62700b98e98fa4c8f4caee6fdc2ea4f3ceb98d2" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "0ff550d0f8892144630fa4e06d03224d36a6aa75" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "529a2d63d2363a9a63454ab122b31dc85d16e8ef"
 :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "d43e6bb05d85e218dcbf8c3fbf158ee633b815da" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "062ec99d21010120892f9c6cb53c1ef2748154bf" :PARSED-COMMENT #) #S(FORGERIE-CORE:COMMIT :SHA "1697be7
f5e9ed16c7290678ef4611b56e5f60ea3" :PARSED-COMMENT #) ...))) :SNIPPETS (#S(FORGERIE-CORE:SNIPPET :ID 1 :TITLE "dockscript.sh" :FILES (#S(FORGERIE-CORE::FILE :ID 5 :NAME "dockscript.sh" :SIZE 621 :LOCATION "/tmp/forgerie/phabricator/files/z
zv/PHID-FILE-ijvbukainvcve35ujzzv" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #8="olasd" :NAME #9="Nicolas Dandrimont" :EMAILS (# #) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME #1# :SIZE
12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE #2#)) :NOTES (#S(FORGERIE-CORE::NOTE :ID "P1" :TEXT # :AUTHOR # :DATE 3650776468)) :DATE 3650776401) #S(FORGERIE-CORE:SNIPPET :ID 2 :TITLE NIL :
FILES (#S(FORGERIE-CORE::FILE :ID 6 :NAME NIL :SIZE 1242 :LOCATION "/tmp/forgerie/phabricator/files/cn6/PHID-FILE-x7e5i3yxuj3qkc6dncn6" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #10="zack" :NAME #11="
Stefano Zacchiroli" :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3650861920) #S(FORGERI
E-CORE:SNIPPET :ID 3 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 7 :NAME NIL :SIZE 965 :LOCATION "/tmp/forgerie/phabricator/files/7gk/PHID-FILE-2wnlhh67hd72mbamq7gk" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER
:USERNAME #10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3650865542) #S(F
ORGERIE-CORE:SNIPPET :ID 4 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 8 :NAME NIL :SIZE 2219 :LOCATION "/tmp/forgerie/phabricator/files/3o2/PHID-FILE-qil2rcesnq2gku3kl3o2" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE
::USER :USERNAME #8# :NAME #9# :EMAILS (# #) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME #1# :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE #2#)) :NOTES NIL :DATE 36508668
15) #S(FORGERIE-CORE:SNIPPET :ID 5 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 10 :NAME NIL :SIZE 445 :LOCATION "/tmp/forgerie/phabricator/files/kvb/PHID-FILE-7xrvc3nxazqsflfwqkvb" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGE
RIE-CORE::USER :USERNAME #10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3
650872498) #S(FORGERIE-CORE:SNIPPET :ID 6 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 12 :NAME NIL :SIZE 655 :LOCATION "/tmp/forgerie/phabricator/files/cf5/PHID-FILE-dir54l6pgfrwouj37cf5" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #
S(FORGERIE-CORE::USER :USERNAME #10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL
:DATE 3650907685) #S(FORGERIE-CORE:SNIPPET :ID 7 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 13 :NAME NIL :SIZE 868 :LOCATION "/tmp/forgerie/phabricator/files/5c6/PHID-FILE-i37jrihwpb2fwxp755c6" :MIMETYPE "text/plain; charset=utf-8")) :A
UTHOR #S(FORGERIE-CORE::USER :USERNAME #10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOT
ES NIL :DATE 3651219510) #S(FORGERIE-CORE:SNIPPET :ID 8 :TITLE "gzip-sha1-files.py" :FILES (#S(FORGERIE-CORE::FILE :ID 15 :NAME NIL :SIZE 1627 :LOCATION "/tmp/forgerie/phabricator/files/hs2/PHID-FILE-nwhmaiq6cltt26mhkhs2" :MIMETYPE "text/p
lain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7
tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3651298031) #S(FORGERIE-CORE:SNIPPET :ID 9 :TITLE "directory entry / list duplication on linux.git" :FILES (#S(FORGERIE-CORE::FILE :ID 17 :NAME "directory entry / list duplication on linux.git" :SIZE 5
08 :LOCATION "/tmp/forgerie/phabricator/files/qhq/PHID-FILE-dvz5gxtxjky57possqhq" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #8# :NAME #9# :EMAILS (# #) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966
263 :NAME #1# :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE #2#)) :NOTES NIL :DATE 3651308475) #S(FORGERIE-CORE:SNIPPET :ID 10 :TITLE "First import of linux.git" :FILES (#S(FORGERIE-CO
RE::FILE :ID 18 :NAME "First import of linux.git" :SIZE 3180 :LOCATION "/tmp/forgerie/phabricator/files/fwm/PHID-FILE-tz5f5kxsgplyubxbvfwm" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #8# :NAME #9# :EMA
ILS (# #) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME #1# :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE #2#)) :NOTES NIL :DATE 3651309746) #S(FORGERIE-CORE:SNIPPET :ID 11
 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 19 :NAME NIL :SIZE 373 :LOCATION "/tmp/forgerie/phabricator/files/4a3/PHID-FILE-fnx3t6vhnglhkhbao4a3" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #10# :NAM
E #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3651386530) #S(FORGERIE-CORE:SNIPPE
T :ID 12 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 21 :NAME NIL :SIZE 692 :LOCATION "/tmp/forgerie/phabricator/files/lwo/PHID-FILE-tduzyqgor7d26ki3ylwo" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME #
10# :NAME #11# :EMAILS (#) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME #5# :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE #6#)) :NOTES NIL :DATE 3651583116) ...) :MERGE-REQ
UESTS (#S(FORGERIE-CORE:MERGE-REQUEST :ID 1495 :VC-REPOSITORY #S(FORGERIE-CORE::VC-REPOSITORY :NAME #12="sysadm-provisioning" :SLUG #13="swh-sysadmin-provisioning" :PRIMARY-PROJECTS NIL :GIT-LOCATION #14="/srv/phabricator/repos/SPRE/" :PRO
JECTS (# #) :PRIVATE NIL :COMMITS NIL) :TITLE "azure/terraform: Add azure vault vm using terraform" :DESCRIPTION ("Related " (:TICKET "1716" "T1716") "
" (:H2 "Test Plan" "
== Test Plan ==
") #<(SIMPLE-ARRAY CHARACTER (9381))

...
7: ((LAMBDA NIL :IN FORGERIE-CORE:RUN))
8: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
9: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-THREAD::RUN))
10: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
11: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::RUN))
12: (SB-THREAD::RUN)
13: ("foreign function: call_into_lisp")
14: ("foreign function: funcall1")

unhandled condition in --disable-debugger mode, quitting
Shutting down....
WARNING:
   Problem running exit hook #<FUNCTION (LAMBDA ()
                                          :IN
                                          FORGERIE-CORE:RUN) {10087F5D6B}>:
  In thread #<THREAD RUNNING {1001890103}>, attempt to join the current thread.
Shutting down....

real    14m54.306s
user    3m54.792s
sys     4m11.098s

[2]

The source of that issue was the ssh connection failing to be established [1]
Fixing with ssh-keygen [2]

Triggering back the migration fails [3] as the repository now exists.
So cleaning it up (in gitlab ui and in forgerie container [4]) then triggering back the migration.

[1]

forgerie@bespin:~$ cat ~/.ssh/config
Host gitlab-staging.swh.network
  User git
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id-migration-gitlab
forgerie@bespin:~$ ssh git@gitlab-staging.swh.network
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:ysZeeO6dabyhp6pTNq07TWWzI8K/+urH//zNcSG5wDY.
Please contact your system administrator.
Add correct host key in /srv/forgerie/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /srv/forgerie/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f "/srv/forgerie/.ssh/known_hosts" -R "gitlab-staging.swh.network"
ECDSA host key for gitlab-staging.swh.network has changed and you have requested strict checking.
Host key verification failed.

[2]

forgerie@bespin:~$ ssh-keygen -f "/srv/forgerie/.ssh/known_hosts" -R "gitlab-staging.swh.network"
# Host gitlab-staging.swh.network found: line 1
/srv/forgerie/.ssh/known_hosts updated.
Original contents retained as /srv/forgerie/.ssh/known_hosts.old
forgerie@bespin:~$ ssh git@gitlab-staging.swh.network
The authenticity of host 'gitlab-staging.swh.network (20.103.140.206)' can't be established.
ECDSA key fingerprint is SHA256:ysZeeO6dabyhp6pTNq07TWWzI8K/+urH//zNcSG5wDY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes                                                                                                                                                                       Warning: Permanently added 'gitlab-staging.swh.network' (ECDSA) to the list of known hosts.                                                                                                                                                    Warning: the ECDSA host key for 'gitlab-staging.swh.network' differs from the key for the IP address '20.103.140.206'                                                                                                                          Offending key for IP in /srv/forgerie/.ssh/known_hosts:1                                                                                                                                                                                       Are you sure you want to continue connecting (yes/no)? ^C                                                                                                                                                                                      forgerie@bespin:~$ ssh-keygen -f "/srv/forgerie/.ssh/known_hosts" -R "20.103.140.206"                                                                                                                                                          # Host 20.103.140.206 found: line 1                                                                                                                                                                                                            /srv/forgerie/.ssh/known_hosts updated.                                                                                                                                                                                                        Original contents retained as /srv/forgerie/.ssh/known_hosts.old                                                                                                                                                                               forgerie@bespin:~$ ssh git@gitlab-staging.swh.network                                                                                                                                                                                          Warning: Permanently added the ECDSA host key for IP address '20.103.140.206' to the list of known hosts.                                                                                                                                      PTY allocation request failed on channel 0                                                                                                                                                                                                     Welcome to GitLab, @root!                                                                                                                                                                                                                      Connection to gitlab-staging.swh.network closed.

[3]

forgerie@bespin:~$ time /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"                                                                                                                                               To load "forgerie":                                                                                                                                                                                                                              Load 1 ASDF system:                                                                                                                                                                                                                              forgerie                                                                                                                                                                                                                                   ; Loading "forgerie"                                                                                                                                                                                                                           .........                                                                                                                                                                                                                                      Unhandled FORGERIE-GITLAB::HTTP-ERROR in thread #<SB-THREAD:THREAD RUNNING                                                                                                                                                                                                                        {10066FC653}>:                                                                                                                                                                                Http error code: 400, resp: (MESSAGE                                                                                                                                                                                                                                        (NAME (has already been taken) PATH                                                                                                                                                                                                             (has already been taken)))                                                                                                                                                                                                                                                                                                                                                                                                                                    Backtrace for: #<SB-THREAD:THREAD RUNNING {10066FC653}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<FORGERIE-GITLAB::HTTP-ERROR {1008E76D13}> #<unused argument> :QUIT T)                                                                                                                                   1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<FORGERIE-GITLAB::HTTP-ERROR {1008E76D13}>)                                                                                                                                                     2: (INVOKE-DEBUGGER #<FORGERIE-GITLAB::HTTP-ERROR {1008E76D13}>)
3: (ERROR #<FORGERIE-GITLAB::HTTP-ERROR {1008E76D13}>)
4: (FORGERIE-GITLAB::MAKE-REQUEST "projects" :POST (("namespace_id" . "4") ("name" . "sysadm-provisioning") ("path" . "swh-sysadmin-provisioning") ("tag_list" . "system_administration,system_administrators,sysadm") ("issues_access_level" .
 #1="enabled") ("visibility" . "public") ("merge_requests_access_level" . #1#)) :SUDO NIL)
5: (FORGERIE-GITLAB::CREATE-PROJECT #S(FORGERIE-CORE::VC-REPOSITORY :NAME "sysadm-provisioning" :SLUG "swh-sysadmin-provisioning" :PRIMARY-PROJECTS NIL :GIT-LOCATION "/srv/phabricator/repos/SPRE/" :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME
 "System administration" :TAGS ("system_administration")) #S(FORGERIE-CORE::PROJECT :NAME "System administrators" :TAGS ("system_administrators" "sysadm"))) :PRIVATE NIL :COMMITS (#S(FORGERIE-CORE:COMMIT :SHA "7b90b55041ccb14ebb15d8ed59c42
d2bb64ebfd3" :PARSED-COMMENT ("Align worker17 hardware spec to worker18

[4]

forgerie@bespin:~$ rm -rf /tmp/forgerie/phabricator/swh-sysadmin-provisioning/ /tmp/forgerie/gitlab/swh-sysadmin-provisioning/

Migration failed again with a problem when creating a snippet [1]

[2]

[1]

forgerie@bespin:~$ time /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"                                                                                                                                     [371/2681]
To load "forgerie":
  Load 1 ASDF system:
    forgerie
; Loading "forgerie"
.........
^O1Unhandled FORGERIE-GITLAB::HTTP-ERROR in thread #<SB-THREAD:THREAD RUNNING
                                                   {1001888103}>:
  Http error code: 400, resp: (MESSAGE
                               (ERROR
                                Repository Error creating the snippet - Invalid file name))

Backtrace for: #<SB-THREAD:THREAD RUNNING {1001888103}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<FORGERIE-GITLAB::HTTP-ERROR {1013CE6D93}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<FORGERIE-GITLAB::HTTP-ERROR {1013CE6D93}>)
2: (INVOKE-DEBUGGER #<FORGERIE-GITLAB::HTTP-ERROR {1013CE6D93}>)
3: (ERROR #<FORGERIE-GITLAB::HTTP-ERROR {1013CE6D93}>)
4: (FORGERIE-GITLAB::MAKE-REQUEST "/projects/2/snippets" :POST (("title" . "Des projets après Inria ? // What’s about after Inria ?") ("content" . #<(SIMPLE-ARRAY CHARACTER (2118))


-------- Forwarded Message --------
Subject: Des projets après Inria ? // What’s about after Inria ?
Date: Mon, 24 Jan 2022 17:35:30 +0100 (CET)
From: Gregoire Maurice <gregoire.maurice@inria.fr... {1013C7C24F}>) ("visibility" . "public") ("file_name" . "Des projets après Inria ? // What’s about after Inria ?")) :SUDO NIL)
5: (FORGERIE-GITLAB::CREATE-SNIPPET #S(FORGERIE-CORE:SNIPPET :ID 1262 :TITLE "Des projets après Inria ? // What’s about after Inria ?" :FILES (#S(FORGERIE-CORE::FILE :ID 5332516 :NAME "Des projets après Inria ? // What’s about after Inria
?" :SIZE 2118 :LOCATION "/tmp/forgerie/phabricator/files/zhl/PHID-FILE-4thcaeg7bkeizqb4szhl" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "vlorentz" :NAME "Valentin Lorentz" :EMAILS (#S(FORGERIE-CORE::EM
AIL :ADDRESS "vlorentz_softwareheritage.org@example.org" :IS-PRIMARY T)) :ADMIN NIL :AVATAR #S(FORGERIE-CORE::FILE :ID 3315393 :NAME "profile" :SIZE 24576 :LOCATION "/tmp/forgerie/phabricator/files/efx/PHID-FILE-6yg2nqica7dzwao3yefx" :MIME
TYPE "image/jpeg")) :NOTES NIL :DATE 3852033412))
6: ((FLET FORGERIE-GITLAB::MAP-WITH-NOTE-MAPPING-CATCH :IN FORGERIE-CORE:EXPORT-FORGE) #<FUNCTION FORGERIE-GITLAB::CREATE-SNIPPET> (#S(FORGERIE-CORE:SNIPPET :ID 1 :TITLE "dockscript.sh" :FILES (#S(FORGERIE-CORE::FILE :ID 5 :NAME "dockscrip
t.sh" :SIZE 621 :LOCATION "/tmp/forgerie/phabricator/files/zzv/PHID-FILE-ijvbukainvcve35ujzzv" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "olasd" :NAME "Nicolas Dandrimont" :EMAILS (#S(FORGERIE-CORE::E
MAIL :ADDRESS "nicolas.dandrimont_gmail.com@example.org" :IS-PRIMARY NIL) #S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME "profile" :SIZE 12136
:LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE "image/jpeg")) :NOTES (#S(FORGERIE-CORE::NOTE :ID "P1" :TEXT (#<(SIMPLE-ARRAY CHARACTER (262)) To be used with a udev config snippet such as:
ACTION=="add", ENV{DEVTYPE}=="usb_device", ENV{ID_MODEL}=="USB5534B", RUN+="/usr/local/bin/dock add"
ACTION=="remove", ENV{DEVTYPE}=="usb_device", EN... {100344DF8F}>) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "olasd" :NAME "Nicolas Dandrimont" :EMAILS # :ADMIN T :AVATAR #) :DATE 3650776468)) :DATE 3650776401) #S(FORGERIE-CORE:SNIPPET :ID
2 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 6 :NAME NIL :SIZE 1242 :LOCATION "/tmp/forgerie/phabricator/files/cn6/PHID-FILE-x7e5i3yxuj3qkc6dncn6" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "zack" :
NAME "Stefano Zacchiroli" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME "profile" :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya
/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE "image/jpeg")) :NOTES NIL :DATE 3650861920) #S(FORGERIE-CORE:SNIPPET :ID 3 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 7 :NAME NIL :SIZE 965 :LOCATION "/tmp/forgerie/phabricator/files/7gk/PHID-F
ILE-2wnlhh67hd72mbamq7gk" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "zack" :NAME "Stefano Zacchiroli" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN T :
AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME "profile" :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7tya" :MIMETYPE "image/jpeg")) :NOTES NIL :DATE 3650865542) #S(FORGERIE-CORE:SNIPPET :ID 4 :TITLE
 NIL :FILES (#S(FORGERIE-CORE::FILE :ID 8 :NAME NIL :SIZE 2219 :LOCATION "/tmp/forgerie/phabricator/files/3o2/PHID-FILE-qil2rcesnq2gku3kl3o2" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "olasd" :NAME "N
icolas Dandrimont" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas.dandrimont_gmail.com@example.org" :IS-PRIMARY NIL) #S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE
::FILE :ID 2966263 :NAME "profile" :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE "image/jpeg")) :NOTES NIL :DATE 3650866815) #S(FORGERIE-CORE:SNIPPET :ID 5 :TITLE NIL :FILES (#S(FORGER
IE-CORE::FILE :ID 10 :NAME NIL :SIZE 445 :LOCATION "/tmp/forgerie/phabricator/files/kvb/PHID-FILE-7xrvc3nxazqsflfwqkvb" :MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "zack" :NAME "Stefano Zacchiroli" :EMA
ILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "zack_upsilon.cc@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 203038 :NAME "profile" :SIZE 16531 :LOCATION "/tmp/forgerie/phabricator/files/tya/PHID-FILE-vc2fhnioi2wwib4o7ty
a" :MIMETYPE "image/jpeg")) :NOTES NIL :DATE 3650872498) #S(FORGERIE-CORE:SNIPPET :ID 6 :TITLE NIL :FILES (#S(FORGERIE-CORE::FILE :ID 12 :NAME NIL :SIZE 655 :LOCATION "/tmp/forgerie/phabricator/files/cf5/PHID-FILE-dir54l6pgfrwouj37cf5" :MI

And now it fails on snippet creation which needs access to the rails console (in the current state of the migration script) [1]
I've got no proper setup which allows this to work now [2].
The only way i see is to comment that part for now... [3]

[2] P1381

[1]

forgerie@bespin:~$ time /opt/forgerie/bin/run | tee "/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"
To load "forgerie":
  Load 1 ASDF system:
    forgerie
; Loading "forgerie"
.........
Unhandled END-OF-FILE in thread #<SB-THREAD:THREAD RUNNING {1004B30003}>:
  end of file on #<SB-SYS:FD-STREAM for "descriptor 8" {100C75ECE3}>

Backtrace for: #<SB-THREAD:THREAD RUNNING {1004B30003}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<END-OF-FILE {100C75F363}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<END-OF-FILE {100C75F363}>)
2: (INVOKE-DEBUGGER #<END-OF-FILE {100C75F363}>)
3: (ERROR END-OF-FILE :STREAM #<SB-SYS:FD-STREAM for "descriptor 8" {100C75ECE3}>)
4: (READ-LINE #<SB-SYS:FD-STREAM for "descriptor 8" {100C75ECE3}> T NIL #<unused argument>)
5: (FORGERIE-GITLAB::RAILS-COMMAND "s = Snippet.find(1)")
6: (FORGERIE-GITLAB::CREATE-SNIPPET #S(FORGERIE-CORE:SNIPPET :ID 1 :TITLE "dockscript.sh" :FILES (#S(FORGERIE-CORE::FILE :ID 5 :NAME "dockscript.sh" :SIZE 621 :LOCATION "/tmp/forgerie/phabricator/files/zzv/PHID-FILE-ijvbukainvcve35ujzzv" :
MIMETYPE "text/plain; charset=utf-8")) :AUTHOR #S(FORGERIE-CORE::USER :USERNAME "olasd" :NAME "Nicolas Dandrimont" :EMAILS (#S(FORGERIE-CORE::EMAIL :ADDRESS "nicolas.dandrimont_gmail.com@example.org" :IS-PRIMARY NIL) #S(FORGERIE-CORE::EMAI
L :ADDRESS "nicolas_dandrimont.eu@example.org" :IS-PRIMARY T)) :ADMIN T :AVATAR #S(FORGERIE-CORE::FILE :ID 2966263 :NAME "profile" :SIZE 12136 :LOCATION "/tmp/forgerie/phabricator/files/463/PHID-FILE-sepmsswq26g5dmbnc463" :MIMETYPE "image/
jpeg")) :NOTES (#S(FORGERIE-CORE::NOTE :ID "P1" :TEXT (#<(SIMPLE-ARRAY CHARACTER (262)) To be used with a udev config snippet such as:

[3]

forgerie@bespin:~$ grep -r "create-snippet" /opt/forgerie/
/opt/forgerie/src/main/gitlab/export.lisp:     ;; (map-with-note-mapping-catch #'create-snippet (getf data :snippets))
/opt/forgerie/src/main/gitlab/export.lisp:(defun create-snippet (snippet)

Ah the migration scripts are still relying on the rails console to actually create merge requests... so we need that setup to work.
So i dug as to why my laptop was ok but not my desktop.

And i turns out i had an extra dns entry in yavin4...

forgerie@bespin:~$ grep euwest /etc/hosts
192.168.200.13 euwest-gitlab-staging-89229db0.dd8c7b66-1c78-4e28-9fe3-6f454883861d.privatelink.westeurope.azmk8s.io
forgerie@bespin:~$ kubectl exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
 GitLab:       15.0.0 (8a186dedfc1) FOSS
 GitLab Shell: 14.3.0
 PostgreSQL:   12.7
------------------------------------------------------------[ booted in 19.68s ]
Loading production environment (Rails 6.1.4.7)
irb(main):001:0> Snippet.find(1)
=> #<ProjectSnippet id:1 migrated/Migration$1>

So maybe that will unstuck it all.

Better cli to execute...

forgerie@bespin:~$ logpath="/tmp/forgerie/run-$(date +%Y%m%d-%H%M).log"; echo "## Running migration logs in $logpath ##"; time /opt/forgerie/bin/run | tee $logpath
## Running migration logs in /tmp/forgerie/run-20220613-1410.log ##
...

So i dug as to why my laptop was ok but not my desktop.

@vsellier reminded me of the following that i had executed a while back in the laptop (yavin4) indeed [1].
Instructions include the /etc/hosts modification (so that explain why it unstuck stuff a bit).
(As for the ~/.kube/config, it's already pushed on bespin).

[1] https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w#Azure-Kubernetes-cluster-provisionning

Status update:

  • Clean up gitlab state
  • Checks
    • make sure host ssh key did not change after state reset (ssh-keyscan)
    • state folder permissions ok
    • access token key generated (and properly referenced in config.lisp)
  • Trigger migration
  • Report migration status
  • Report migration status

Fail [1] on the rails command execution again.
Actually troubleshooting...
and, after activating the debug option [2], finally something actionable [3]

It's not happy about our rails-command-args setup... [4]

[2]

(setf forgerie-core:*debug* t)

[3]

...
Booting: Error: unknown command "exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console" for "kubectl"
Booting: Run 'kubectl --help' for usage.
Unhandled END-OF-FILE in thread #<SB-THREAD:THREAD RUNNING {1004B30003}>:
  end of file on #<SB-SYS:FD-STREAM for "descriptor 8" {10034B89B3}>
...

[4]

(setf forgerie-gitlab:*rails-command* "/usr/bin/kubectl")
(setf forgerie-gitlab:*rails-command-args* '("exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console"))

[1]

Changing the command args from [1] to [2], the run cli no longer complains! [3]

[3]

Booting: Unable to use a TTY - input is not a terminal or the right kind of file
Booting: --------------------------------------------------------------------------------
Booting:  Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
Booting:  GitLab:       15.0.0 (8a186dedfc1) FOSS
Booting:  GitLab Shell: 14.3.0
Booting:  PostgreSQL:   12.7
Booting: ------------------------------------------------------------[ booted in 16.11s ]

[1]

(setf forgerie-gitlab:*rails-command-args* '("exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console"))

[2]

(setf forgerie-gitlab:*rails-command-args* '("exec" "-ti" "-n" "gitlab-system" "deployment/gitlab-toolbox" "--" "/srv/gitlab/bin/rails" "console"))

And it seems it finished without any issues this time [1] [2]

[1]

Shutting down....

real    14m3.534s
user    0m35.738s
sys     0m25.314s

[2] full logs

Note:

  • Private repository after migration are private.

I've triggered the remaining sysadm repositories to migrate.

ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)

The migration did a lot more work this time. It finally crashed on the git am command though [1] [2] [3].

[1] man git am says: 'git-am - Apply a series of patches from a mailbox'

$ grep "//working.patch" archives/run-20220615-1235-all-sysadm-repositories.log | head -1
  Got error running git command am with args ("/tmp/forgerie/gitlab//working.patch") in dir /tmp/forgerie/gitlab/puppet-swh-site

[2] with a tiny bit more context

Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD RUNNING {10068A0793}>:
  Got error running git command am with args ("/tmp/forgerie/gitlab//working.patch") in dir /tmp/forgerie/gitlab/puppet-swh-site

Backtrace for: #<SB-THREAD:THREAD RUNNING {10068A0793}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1017080693}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1017080693}>)
2: (INVOKE-DEBUGGER #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1017080693}>)
3: (ERROR "Got error running git command ~A with args ~S in dir ~A" "am" ("/tmp/forgerie/gitlab//working.patch") "/tmp/forgerie/gitlab/puppet-swh-site")
4: (FORGERIE-CORE:GIT-CMD "/tmp/forgerie/gitlab/puppet-swh-site" "am" ("/tmp/forgerie/gitlab//working.patch") :ERROR T :DEBUG NIL :INPUT NIL)
5: (FORGERIE-GITLAB::CREATE-MERGE-REQUEST #S(FORGERIE-CORE:MERGE-REQUEST :ID 6686 :VC-REPOSITORY #S(FORGERIE-CORE::VC-REPOSITORY :NAME "puppet-swh-site" :SLUG "puppet-swh-site" :PRIMARY-PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Puppet rec
ipes" :TAGS ("puppet_recipes" "puppet"))) :GIT-LOCATION "/srv/phabricator/repos/SPSITE/" :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Language-Puppet" :TAGS ("language-puppet")) #S(FORGERIE-CORE::PROJECT :NAME "System administration" :TAGS
("system_administration")) #S(FORGERIE-CORE::PROJECT :NAME "Sync to GitHub" :TAGS ("sync_to_github"))) :PRIVATE NIL :COMMITS NIL) :TITLE "hedgedoc: Drop unnecessary and breaking dependencies" :DESCRIPTION ("With the following package decla
ration, puppet fails to apply. They are unecessary as
they are not installed (or pulled as dependencies) on the node without impacting the
runtime.
...
real    484m15.340s
user    194m42.485s
sys     255m37.446s

[3] full logs

Unstuck after some mob debugging ;)
There is a fallback which tries and use the git am command and fails as some git config is missing.
We unstuck it by adding some .gitconfig in the dockerfile declaration.

But note that the applied commit is messing up the original author in the commit so it's not perfect. It has the advantage to no longer failing the migration so fine.

/me grunts
another one bites the dust:

                                                   {1001888103}>:
  Http error code: 400, resp: (MESSAGE
                               (AVATAR
                                (file format is not supported. Please try one of the following supported formats: png, jpg, jpeg, gif, bmp, tiff, ico, webp)))

Backtrace for: #<SB-THREAD:THREAD RUNNING {1001888103}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<FORGERIE-GITLAB::HTTP-ERROR {10064C1263}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<FORGERIE-GITLAB::HTTP-ERROR {10064C1263}>)
2: (INVOKE-DEBUGGER #<FORGERIE-GITLAB::HTTP-ERROR {10064C1263}>)
3: (ERROR #<FORGERIE-GITLAB::HTTP-ERROR {10064C1263}>)
4: (FORGERIE-GITLAB::MAKE-REQUEST "users" :POST (("name" . "Nicolas Dandrimont") ("email" . "nicolas_dandrimont.eu@example.org") ("admin" . #1="true") ("reset_password" . #1#) ("username" . "olasd") ("avatar" . #P"/tmp/forgerie/phabricator
/files/463/PHID-FILE-sepmsswq26g5dmbnc463")) :SUDO NIL)

After unstucking the migration again, he avatar upload routine is now compliant with the
last http client optimization change and works! I've rebased the commit again and pushed
in the swh branch [1].

The migration was done and it was time to trigger it back from scratch to have a
non-failure and timed scenario ok (to have a rough estimation of the time it can be left
unattended and doing the job).

I'm now stuck in reseting the migration state which is now blocked for unknown reason
(for now). I've summarized that blocking point in T4347.

[1] At some point, i'll have to attend to proposing upstream all those commits (as merge
requests) to ease reproductibility migration runs.

A new one...

Got error running git command push with args ("gitlab" "--delete"
                                              "generated-differential-D1804-source") in dir /tmp/forgerie/gitlab/swh-jenkins-jobs

Troubleshooting it:

forgerie@bespin:/tmp/forgerie/gitlab/swh-jenkins-jobs$ git push gitlab --delete generated-differential-D1804-source
Warning: Permanently added 'gitlab-staging.swh.network,20.103.140.206' (ECDSA) to the list of known hosts.
error: unable to delete 'generated-differential-D1804-source': remote ref does not exist
error: failed to push some refs to 'gitlab-staging.swh.network:migrated/swh-jenkins-jobs.git'

My tryout in T4334 made me reset an important file which now messes up everything.
I'm reseting the gitlab state again and doing a migration from scratch.

The last one roughly took 200min (in multiple steps though) so it's fast enough [1]

[1]

real    39m24.705s
real    122m27.491s
real    40m42.343s

(+ 40 39 122) ; 201 min

I'm reseting the gitlab state again and doing a migration from scratch.

That migration actually run without issues and without being human managed [1].

[1]

Plenty of fixes happened again.
Multiple iterations happened on bespin to further check the resulting scripts.

And now migration is ongoing for production with the sysadm repository.
Maybe we can close this now.

@olasd @vsellier what do you think?

ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.