Page MenuHomeSoftware Heritage

Github incremental lister is broken
Closed, MigratedEdits Locked

Description

The following error is reported while trying to execute it in the docker environment:

swh-lister_1                  | [2019-09-17 09:00:10,177: INFO/MainProcess] Received task: swh.lister.github.tasks.IncrementalGitHubLister[ef46a79b-ae47-4897-b8e8-12b84f394d86]  
swh-lister_1                  | [2019-09-17 09:00:10,181: DEBUG/ForkPoolWorker-1] Loading config from lister_github
swh-lister_1                  | [2019-09-17 09:00:10,182: INFO/ForkPoolWorker-1] Loading config file /lister.yml
swh-lister_1                  | [2019-09-17 09:00:10,197: DEBUG/ForkPoolWorker-1] <swh.lister.github.lister.GitHubLister object at 0x7ff65ef802d0> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://swh-storage:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://swh-scheduler-api:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql://postgres@swh-listers-db/swh-listers'}}, 'celery': {'task_broker': 'amqp://guest:guest@amqp//', 'task_modules': ['swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.npm.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks'], 'task_queues': ['swh.lister.bitbucket.tasks.FullBitBucketRelister', 'swh.lister.bitbucket.tasks.IncrementalBitBucketLister', 'swh.lister.bitbucket.tasks.RangeBitBucketLister', 'swh.lister.bitbucket.tasks.ping', 'swh.lister.cgit.tasks.CGitListerTask', 'swh.lister.cgit.tasks.ping', 'swh.lister.cran.tasks.CRANListerTask', 'swh.lister.cran.tasks.ping', 'swh.lister.debian.tasks.DebianListerTask', 'swh.lister.debian.tasks.ping', 'swh.lister.github.tasks.FullGitHubRelister', 'swh.lister.github.tasks.IncrementalGitHubLister', 'swh.lister.github.tasks.RangeGitHubLister', 'swh.lister.github.tasks.ping', 'swh.lister.gitlab.tasks.FullGitLabRelister', 'swh.lister.gitlab.tasks.IncrementalGitLabLister', 'swh.lister.gitlab.tasks.RangeGitLabLister', 'swh.lister.gitlab.tasks.ping', 'swh.lister.gnu.tasks.GNUListerTask', 'swh.lister.gnu.tasks.ping', 'swh.lister.npm.tasks.NpmIncrementalListerTask', 'swh.lister.npm.tasks.NpmListerTask', 'swh.lister.npm.tasks.ping', 'swh.lister.packagist.tasks.PackagistListerTask', 'swh.lister.packagist.tasks.ping', 'swh.lister.phabricator.tasks.FullPhabricatorLister', 'swh.lister.phabricator.tasks.IncrementalPhabricatorLister', 'swh.lister.phabricator.tasks.ping', 'swh.lister.pypi.tasks.PyPIListerTask', 'swh.lister.pypi.tasks.ping']}, 'credentials': {'phabricator': {'forge.softwareheritage.org': [{'password': 'cli-qivc246lczzgbin36q2i4r644mw3'}], 'phabricator.kde.org': [{'password': 'api-zzdqkxwbhq5qilcsz4xykgwupsdc'}]}}, 'cache_responses': False, 'cache_dir': '/srv/softwareheritage/.cache/swh/lister/github'}
swh-lister_1                  | [2019-09-17 09:00:10,250: ERROR/ForkPoolWorker-1] Task swh.lister.github.tasks.IncrementalGitHubLister[ef46a79b-ae47-4897-b8e8-12b84f394d86] raised unexpected: TypeError('%d format: a number is required, not str')
swh-lister_1                  | Traceback (most recent call last):
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
swh-lister_1                  |     R = retval = fun(*args, **kwargs)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 45, in __call__
swh-lister_1                  |     return super().__call__(*args, **kwargs)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
swh-lister_1                  |     return self.run(*args, **kwargs)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/github/tasks.py", line 19, in list_github_incremental
swh-lister_1                  |     lister.run(min_bound=lister.db_last_index(), max_bound=None)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/indexing_lister.py", line 234, in run
swh-lister_1                  |     for i in ingest_indexes():
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/indexing_lister.py", line 215, in ingest_indexes
swh-lister_1                  |     response, injected_repos = self.ingest_data(index)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/lister_base.py", line 482, in ingest_data
swh-lister_1                  |     response = self.safely_issue_request(identifier)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/lister_base.py", line 283, in safely_issue_request
swh-lister_1                  |     r = self.transport_request(identifier)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/lister_transports.py", line 189, in transport_request
swh-lister_1                  |     return self._transport_action(identifier)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/lister_transports.py", line 161, in _transport_action
swh-lister_1                  |     path = self.request_uri(identifier)
swh-lister_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/lister/core/lister_transports.py", line 98, in request_uri
swh-lister_1                  |     path = self.PATH_TEMPLATE % identifier
swh-lister_1                  | TypeError: %d format: a number is required, not str