As in production settings for the objstorage reverse proxy, set HTTP read
timeout to 1h in order to avoid 504 error (Gateway Time-out) when posting
a large amount of contents to the objects storage.
I got hit by that issue when attempting to load the huge subversion repository
for TexLive (svn://tug.org/texlive).
docker-swh-loader-1 | [2022-10-28 01:58:32,371: DEBUG/ForkPoolWorker-1] rev: 90, swhrev: 041c10306f3efb9ad37e77eea5cca2e9772ec5ef, dir: b3e98625596f907e969400f1f4c133a2a397e68c
docker-swh-loader-1 | [2022-10-28 01:58:32,401: DEBUG/ForkPoolWorker-1] Flushing 17021 objects of type content (223127418 bytes)
docker-swh-loader-1 | [2022-10-28 01:59:33,719: ERROR/ForkPoolWorker-1] Loading failure, updating to `failed` status
docker-swh-loader-1 | Traceback (most recent call last):
docker-swh-loader-1 | File "/tmp/tmp.E2SMlP3pbi/swh-loader-core/swh/loader/core/loader.py", line 414, in load
docker-swh-loader-1 | self.store_data()
docker-swh-loader-1 | File "/tmp/tmp.E2SMlP3pbi/swh-loader-svn/swh/loader/svn/loader.py", line 471, in store_data
docker-swh-loader-1 | self.storage.content_add(self._contents)
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 159, in content_add
docker-swh-loader-1 | keys=["sha1", "sha1_git", "sha256", "blake2s256"],
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 230, in object_add
docker-swh-loader-1 | return self.flush()
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 292, in flush
docker-swh-loader-1 | stats = add_fn(list(batch))
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/filter.py", line 58, in content_add
docker-swh-loader-1 | [x for x in content if x.sha256 in contents_to_add]
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add
docker-swh-loader-1 | return self._post("content/add", {"content": content})
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 283, in _post
docker-swh-loader-1 | return self._decode_response(response)
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 365, in _decode_response
docker-swh-loader-1 | self.raise_for_status(response)
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status
docker-swh-loader-1 | super().raise_for_status(response)
docker-swh-loader-1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 355, in raise_for_status
docker-swh-loader-1 | raise exception from None
docker-swh-loader-1 | swh.core.api.RemoteException: <html>
docker-swh-loader-1 | <head><title>504 Gateway Time-out</title></head>
docker-swh-loader-1 | <body>
docker-swh-loader-1 | <center><h1>504 Gateway Time-out</h1></center>
docker-swh-loader-1 | <hr><center>nginx/1.23.2</center>
docker-swh-loader-1 | </body>
docker-swh-loader-1 | </html>