Some HTTP download requests might be throttled by remote servers
so add retry mechanism with exponential backoff to fix tarball
downloads in some loaders.
I observed that kind of issues with the AUR loader when testing
it in docker, see below:
docker-swh-loader-1 | [2022-09-05 13:38:26,727: ERROR/ForkPoolWorker-74] Failed to load branch releases/1.79-1/nvi.tar.gz for https://aur.archlinux.org/packages/nvi docker-swh-loader-1 | Traceback (most recent call last): docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 649, in load docker-swh-loader-1 | res = self._load_release(p_info, origin) docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 822, in _load_release docker-swh-loader-1 | dl_artifacts = self.download_package(p_info, tmpdir) docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 399, in download_package docker-swh-loader-1 | return [download(p_info.url, dest=tmpdir, filename=p_info.filename)] docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/utils.py", line 109, in download docker-swh-loader-1 | ValueError: Fail to query 'https://aur.archlinux.org/cgit/aur.git/snapshot/nvi.tar.gz'. Reason: 429 docker-swh-loader-1 | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] default version: 1.79-1 docker-swh-loader-1 | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] extra branches: {} docker-swh-loader-1 | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] releases: {'1.79-1': []} docker-swh-loader-1 | [2022-09-05 13:38:26,728: DEBUG/ForkPoolWorker-74] snapshot: {'branches': {}} docker-swh-loader-1 | [2022-09-05 13:38:26,729: DEBUG/ForkPoolWorker-74] snapshot: Snapshot(branches=ImmutableDict({}), id=hash_to_bytes('1a8893e6a86f444e8be8e7bda6cb34fb1735a00e')) docker-swh-loader-1 | [2022-09-05 13:38:26,729: DEBUG/ForkPoolWorker-74] Flushing 1 objects of type snapshot docker-swh-loader-1 | [2022-09-05 13:38:26,823: WARNING/ForkPoolWorker-74] 1 failed branches docker-swh-loader-1 | [2022-09-05 13:38:26,823: WARNING/ForkPoolWorker-74] Failed branches: releases/1.79-1/nvi.tar.gz docker-swh-loader-1 | [2022-09-05 13:38:26,827: INFO/ForkPoolWorker-74] Task swh.loader.package.aur.tasks.LoadAur[8fafddd4-9005-4609-a385-c02e546ce52e] succeeded in 0.3548696079997171s: {'status': 'uneventful', 'snapshot_id': '1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'} docker-swh-loader-1 | [2022-09-05 13:38:26,847: DEBUG/ForkPoolWorker-74] Loading config file /loader.yml docker-swh-loader-1 | [2022-09-05 13:38:26,852: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[1b7cae47-9cc1-42e8-b5e8-84231efce0e8] received docker-swh-loader-1 | [2022-09-05 13:38:26,915: DEBUG/ForkPoolWorker-74] last snapshot: None docker-swh-loader-1 | [2022-09-05 13:38:26,916: DEBUG/ForkPoolWorker-74] package_info: AurPackageInfo(url='https://aur.archlinux.org/cgit/aur.git/snapshot/libomxil-component-xvideo.tar.gz', filename='libomxil-component-xvideo.tar.gz', directory_extrinsic_metadata=[], name='libomxil-component-xvideo', version='0.1-1', last_modified='2015-11-27T07:01:23+00:00') docker-swh-loader-1 | [2022-09-05 13:38:27,068: ERROR/ForkPoolWorker-74] Failed to load branch releases/0.1-1/libomxil-component-xvideo.tar.gz for https://aur.archlinux.org/packages/libomxil-component-xvideo docker-swh-loader-1 | Traceback (most recent call last): docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 649, in load docker-swh-loader-1 | res = self._load_release(p_info, origin) docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 822, in _load_release docker-swh-loader-1 | dl_artifacts = self.download_package(p_info, tmpdir) docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/loader.py", line 399, in download_package docker-swh-loader-1 | return [download(p_info.url, dest=tmpdir, filename=p_info.filename)] docker-swh-loader-1 | File "/src/swh-loader-core/swh/loader/package/utils.py", line 109, in download docker-swh-loader-1 | ValueError: Fail to query 'https://aur.archlinux.org/cgit/aur.git/snapshot/libomxil-component-xvideo.tar.gz'. Reason: 429 docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] default version: 0.1-1 docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] extra branches: {} docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] releases: {'0.1-1': []} docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] snapshot: {'branches': {}} docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] snapshot: Snapshot(branches=ImmutableDict({}), id=hash_to_bytes('1a8893e6a86f444e8be8e7bda6cb34fb1735a00e')) docker-swh-loader-1 | [2022-09-05 13:38:27,069: DEBUG/ForkPoolWorker-74] Flushing 1 objects of type snapshot docker-swh-loader-1 | [2022-09-05 13:38:27,116: WARNING/ForkPoolWorker-74] 1 failed branches docker-swh-loader-1 | [2022-09-05 13:38:27,116: WARNING/ForkPoolWorker-74] Failed branches: releases/0.1-1/libomxil-component-xvideo.tar.gz docker-swh-loader-1 | [2022-09-05 13:38:27,118: INFO/ForkPoolWorker-74] Task swh.loader.package.aur.tasks.LoadAur[5ce421cd-1610-4554-b0a4-26b4791145d3] succeeded in 0.2711032420011179s: {'status': 'uneventful', 'snapshot_id': '1a8893e6a86f444e8be8e7bda6cb34fb1735a00e'}