Page MenuHomeSoftware Heritage
Create Task
Maniphest T3211

staging: git loader can't load repositories hosted by github.com
Closed, MigratedEdits Locked
Actions

Description

it seems the port 9418 is filtered somewhere on the path to github (our firewall is not logging anything)

swhworker@worker0:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.com/softwareheritage/swh-core
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/softwareheritage/swh-core' with type 'git'
ERROR:swh.loader.git.loader.GitLoader:Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 333, in load
    more_data_to_fetch = self.fetch_data()
  File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 249, in fetch_data
    self.origin.url, self.base_snapshot, do_progress
  File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 196, in fetch_pack_from_origin
    progress=do_activity,
  File "/usr/lib/python3/dist-packages/dulwich/client.py", line 811, in fetch_pack
    proto, can_read, stderr = self._connect(b'upload-pack', path)
  File "/usr/lib/python3/dist-packages/dulwich/client.py", line 933, in _connect
    raise err
  File "/usr/lib/python3/dist-packages/dulwich/client.py", line 925, in _connect
    s.connect(sockaddr)
TimeoutError: [Errno 110] Connection timed out
{'status': 'failed'}

Event Timeline

vsellier changed the task status from Open to Work in Progress.Apr 6 2021, 2:01 PM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier added a comment.EditedApr 6 2021, 3:19 PM

The wrong network profile was asked for the staging gateway so it seems it doesn't have a complete access to internet.
A mail was sent to the DSI to request an unfiltered access.

vsellier closed this task as Resolved.Apr 6 2021, 7:11 PM

The requested change was make by the DSI. Everything is working well now.

swhworker@worker0:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.com/softwareheritage/swh-core
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/softwareheritage/swh-core' with type 'git'
Enumerating objects: 369, done.
Counting objects: 100% (369/369), done.
Compressing objects: 100% (211/211), done.
Total 4986 (delta 244), reused 257 (delta 151), pack-reused 4617
INFO:swh.loader.git.loader.GitLoader:Listed 384 refs for repo https://github.com/softwareheritage/swh-core
{'status': 'eventful'}

real	0m9.579s
user	0m2.732s
sys	0m0.101s
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:02 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal