This is staging but nonetheless, that could happen in production.
Currently this error occured during ingestion of a deposit:
Feb 24 20:36:42 worker0 python3[657402]: [2021-02-24 20:36:42,387: INFO/MainProcess] Received task: swh.loader.package.deposit.tasks.LoadDeposit[e3d3bc13-4ce4-4d5d-b6de-aa41f561cba3]
Feb 24 20:36:43 worker0 python3[657407]: [2021-02-24 20:36:43,888: ERROR/ForkPoolWorker-1] Failed loading branch HEAD for https://doi.org/10.5281/6a78d227-ae11-4b78-be69-100ba7faf725
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 426, in load
res = self._load_revision(p_info, origin)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 541, in _load_revision
dl_artifacts = self.download_package(p_info, tmpdir)
File "/usr/lib/python3/dist-packages/swh/loader/package/deposit/loader.py", line 186, in download_package
return [self.client.archive_get(self.deposit_id, tmpdir, p_info.filename)]
File "/usr/lib/python3/dist-packages/swh/loader/package/deposit/loader.py", line 336, in archive_get
return download(url, dest=tmpdir, filename=filename, auth=self.auth)
File "/usr/lib/python3/dist-packages/swh/loader/package/utils.py", line 81, in download
raise ValueError("Fail to query '%s'. Reason: %s" % (url, response.status_code))
ValueError: Fail to query 'https://deposit-rp.internal.staging.swh.network/1/private/98/raw/'. Reason: 500
Feb 24 20:36:45 worker0 python3[657407]: [2021-02-24 20:36:45,179: WARNING/ForkPoolWorker-1] 1 failed branches
Feb 24 20:36:45 worker0 python3[657407]: [2021-02-24 20:36:45,180: WARNING/ForkPoolWorker-1] Failed branches: HEAD[1] https://sentry.softwareheritage.org/share/issue/d7f7880e5d874498b7ff147cbca9d4c4/
Deposit information:
swh-deposit=> \conninfo You are connected to database "swh-deposit" as user "guest" on host "db1.internal.staging.swh.network" (address "192.168.130.11") at port "5432". SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) swh-deposit=> select * from deposit where id = 98; -[ RECORD 1 ]--+------------------------------------------------------------- id | 98 reception_date | 2021-02-24 20:35:25.262905+00 complete_date | 2021-02-24 20:36:11.063726+00 external_id | swhid | status | verified client_id | 9 collection_id | 8 parent_id | status_detail | swhid_context | check_task_id | 18736121 load_task_id | 18736136 origin_url | https://doi.org/10.5281/6a78d227-ae11-4b78-be69-100ba7faf725
Associated task is marked as completed:
swh-scheduler=> select * from task where id=18736136;
-[ RECORD 1 ]----+------------------------------------------------------------------------------------------------------------------
id | 18736136
type | load-deposit
arguments | {"args": [], "kwargs": {"url": "https://doi.org/10.5281/6a78d227-ae11-4b78-be69-100ba7faf725", "deposit_id": 98}}
next_run | 2021-02-24 20:36:30.43889+00
current_interval | 1 day
status | completed
policy | oneshot
retries_left | 3
priority |There are 2 problems:
- the deposit should be marked as "failed" state
- the actual loading issue