Page MenuHomeSoftware Heritage

Add script to mirror an upstream repository to our forge
ClosedPublic

Authored by ardumont on May 18 2022, 2:42 PM.

Details

Summary

This reuses the new common library [1] to mirror a new phabricator repo (e.g [2]).

This:

  • creates a new repository in our forge
  • clones locally a new repository
  • pushes the local clone to the new swh forge repository
  • configures the remote properly

An example of mirrored repository bootstraped with this.

Related to T2225
[1] Depends on D7847

[2] https://forge.softwareheritage.org/source/forgerie/

Test Plan
$ workon swh
$ python mirror-upstream-repo https://code.librehq.com/ots/forgerie/
Repository 263 still importing, sleeping for 0.5 seconds...
Repository 263 still importing, sleeping for 1.0 seconds...
Repository 263 still importing, sleeping for 2.0 seconds...
Repository 263 still importing, sleeping for 4.0 seconds...
... some failure here ...
...
$ python bin/mirror-upstream-repo --clone https://code.librehq.com/ots/forgerie/ --repo-id 263
Cloning into 'forgerie'...
warning: You appear to have cloned an empty repository.
warning: redirecting to https://code.librehq.com/ots/forgerie.git/
remote: Enumerating objects: 957, done.
remote: Counting objects: 100% (253/253), done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 957 (delta 137), reused 94 (delta 52), pack-reused 704
Receiving objects: 100% (957/957), 156.04 KiB | 739.00 KiB/s, done.
Resolving deltas: 100% (539/539), done.
From https://code.librehq.com/ots/forgerie
 * [new branch]      bug-trackers-ph-gl-mapping -> upstream/bug-trackers-ph-gl-mapping
 * [new branch]      main                       -> upstream/main
error: pathspec 'master' did not match any file(s) known to git
Traceback (most recent call last):
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 63, in <module>
    main()
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 53, in main
    clone_and_push_repo(
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/common.py", line 92, in clone_and_push_repo
    subprocess.check_call(["git", "checkout", default_branch])
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'checkout', 'master']' returned non-zero exit status 1.
$ python bin/mirror-upstream-repo --clone https://code.librehq.com/ots/forgerie/ --repo-id 263 --default-branch main
fatal: destination path 'forgerie' already exists and is not an empty directory.
Traceback (most recent call last):
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 63, in <module>
    main()
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 53, in main
    clone_and_push_repo(
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/common.py", line 84, in clone_and_push_repo
    subprocess.check_call(["git", "clone", phabricator_repo, module_name])
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://forge.softwareheritage.org/source/forgerie.git', 'forgerie']' returned non-zero exit status 128.
$ python bin/mirror-upstream-repo --no-clone https://code.librehq.com/ots/forgerie/ --repo-id 263 --default-branch main
error: remote upstream already exists.
warning: redirecting to https://code.librehq.com/ots/forgerie.git/
branch 'main' set up to track 'upstream/main'.
Switched to a new branch 'main'
Already up to date.
Enumerating objects: 954, done.
Counting objects: 100% (954/954), done.
Delta compression using up to 8 threads
Compressing objects: 100% (312/312), done.
Writing objects: 100% (954/954), 155.29 KiB | 155.29 MiB/s, done.
Total 954 (delta 539), reused 954 (delta 539), pack-reused 0
remote: Resolving deltas: 100% (539/539), done.
To forge.softwareheritage.org:/source/forgerie.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

Diff Detail

Repository
rSENV Puppet Environment
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont edited the test plan for this revision. (Show Details)
ardumont retitled this revision from scripts: Extract reusable functions to a common module to Add script to mirror an upstream repository to our forge.May 18 2022, 2:47 PM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Actual commit message amending

This revision is now accepted and ready to land.May 19 2022, 4:31 PM