Change Details

swh-scheduler-runner_1 | INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | accepting AMQP connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672) amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/' swh-loader_1 | [2020-10-26 14:58:45,438: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[567171db-a00c-4645-805c-61bf0c17b251] swh-loader_1 | [2020-10-26 14:58:45,468: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/add_multi swh-scheduler-runner_1 | INFO:swh.scheduler.cli.admin.runner:Scheduled 1 tasks swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/visit/add swh-loader_1 | [2020-10-26 14:58:47,604: INFO/ForkPoolWorker-1] Load origin 'https://github.com/python/cpython' with type 'git' swh-storage_1 | [2020-10-26 14:58:47 +0000] [43] [DEBUG] POST /origin/visit/get_latest Enumerating objects: 720, done. | Enumerating objects: 1077409 Counting objects: 100% (720/720), done.ing objects: 0% (1/720) Compressing objects: 100% (322/322), done.ing objects: 0% (1/322) swh-loader_1 | Total 1078129 (delta 553), reused 399 (delta 398), pack-reused 1077409 swh-loader_1 | [2020-10-26 15:00:03,431: INFO/ForkPoolWorker-1] Listed 23442 refs for repo https://github.com/python/cpython swh-storage_1 | [2020-10-26 15:05:38 +0000] [43] [DEBUG] POST /content/missing kafka_1 | [2020-10-26 15:05:39,388] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:51 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:04 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:17 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:43 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:03 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:06 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:15 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:47 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:06 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:18 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:30 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:39 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:27 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:36 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:49 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:54 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:30 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:45 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | ERROR:root:An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | [2020-10-26 15:13:13,789] ERROR in app: Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:swh.storage.api.server:Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:root:500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-loader_1 | [2020-10-26 15:13:13,841: ERROR/ForkPoolWorker-1] Loading failure, updating to `partial` status swh-loader_1 | Traceback (most recent call last): swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 319, in load swh-loader_1 | self.store_data() swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 418, in store_data swh-loader_1 | self.storage.content_add([obj]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 97, in content_add swh-loader_1 | return self.flush(["content"]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 137, in flush swh-loader_1 | stats = add_fn(list(batch)) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/filter.py", line 43, in content_add swh-loader_1 | [x for x in content if x.sha256 in contents_to_add] swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add swh-loader_1 | return self.post("content/add", {"content": content}) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 277, in post swh-loader_1 | return self._decode_response(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 351, in _decode_response swh-loader_1 | self.raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status swh-loader_1 | super().raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 341, in raise_for_status swh-loader_1 | raise exception from None swh-loader_1 | swh.core.api.RemoteException: <RemoteException 500 InternalServerError: []> swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /origin/visit_status/add swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] Closing connection.

$ doco exec swh-scheduler swh scheduler task add load-git -p oneshot url=https://github.com/python/cpython $ doco logs -f --tail 0 | grep -vE "heartbeat|statsd" swh-scheduler-runner_1 | INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | accepting AMQP connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672) amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/' swh-loader_1 | [2020-10-26 14:58:45,438: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[567171db-a00c-4645-805c-61bf0c17b251] swh-loader_1 | [2020-10-26 14:58:45,468: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/add_multi swh-scheduler-runner_1 | INFO:swh.scheduler.cli.admin.runner:Scheduled 1 tasks swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/visit/add swh-loader_1 | [2020-10-26 14:58:47,604: INFO/ForkPoolWorker-1] Load origin 'https://github.com/python/cpython' with type 'git' swh-storage_1 | [2020-10-26 14:58:47 +0000] [43] [DEBUG] POST /origin/visit/get_latest Enumerating objects: 720, done. | Enumerating objects: 1077409 Counting objects: 100% (720/720), done.ing objects: 0% (1/720) Compressing objects: 100% (322/322), done.ing objects: 0% (1/322) swh-loader_1 | Total 1078129 (delta 553), reused 399 (delta 398), pack-reused 1077409 swh-loader_1 | [2020-10-26 15:00:03,431: INFO/ForkPoolWorker-1] Listed 23442 refs for repo https://github.com/python/cpython swh-storage_1 | [2020-10-26 15:05:38 +0000] [43] [DEBUG] POST /content/missing kafka_1 | [2020-10-26 15:05:39,388] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:51 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:04 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:17 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:43 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:03 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:06 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:15 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:47 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:06 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:18 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:30 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:39 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:27 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:36 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:49 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:54 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:30 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:45 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | ERROR:root:An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | [2020-10-26 15:13:13,789] ERROR in app: Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:swh.storage.api.server:Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:root:500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-loader_1 | [2020-10-26 15:13:13,841: ERROR/ForkPoolWorker-1] Loading failure, updating to `partial` status swh-loader_1 | Traceback (most recent call last): swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 319, in load swh-loader_1 | self.store_data() swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 418, in store_data swh-loader_1 | self.storage.content_add([obj]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 97, in content_add swh-loader_1 | return self.flush(["content"]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 137, in flush swh-loader_1 | stats = add_fn(list(batch)) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/filter.py", line 43, in content_add swh-loader_1 | [x for x in content if x.sha256 in contents_to_add] swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add swh-loader_1 | return self.post("content/add", {"content": content}) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 277, in post swh-loader_1 | return self._decode_response(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 351, in _decode_response swh-loader_1 | self.raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status swh-loader_1 | super().raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 341, in raise_for_status swh-loader_1 | raise exception from None swh-loader_1 | swh.core.api.RemoteException: <RemoteException 500 InternalServerError: []> swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /origin/visit_status/add swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] Closing connection.

$ doco exec swh-scheduler swh scheduler task add load-git -p oneshot url=https://github.com/python/cpython $ doco logs -f --tail 0 | grep -vE "heartbeat|statsd" swh-scheduler-runner_1 | INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | accepting AMQP connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672) amqp_1 | amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 === amqp_1 | connection <0.1282.0> (172.19.0.12:50804 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/' swh-loader_1 | [2020-10-26 14:58:45,438: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[567171db-a00c-4645-805c-61bf0c17b251] swh-loader_1 | [2020-10-26 14:58:45,468: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/add_multi swh-scheduler-runner_1 | INFO:swh.scheduler.cli.admin.runner:Scheduled 1 tasks swh-storage_1 | [2020-10-26 14:58:45 +0000] [43] [DEBUG] POST /origin/visit/add swh-loader_1 | [2020-10-26 14:58:47,604: INFO/ForkPoolWorker-1] Load origin 'https://github.com/python/cpython' with type 'git' swh-storage_1 | [2020-10-26 14:58:47 +0000] [43] [DEBUG] POST /origin/visit/get_latest Enumerating objects: 720, done. | Enumerating objects: 1077409 Counting objects: 100% (720/720), done.ing objects: 0% (1/720) Compressing objects: 100% (322/322), done.ing objects: 0% (1/322) swh-loader_1 | Total 1078129 (delta 553), reused 399 (delta 398), pack-reused 1077409 swh-loader_1 | [2020-10-26 15:00:03,431: INFO/ForkPoolWorker-1] Listed 23442 refs for repo https://github.com/python/cpython swh-storage_1 | [2020-10-26 15:05:38 +0000] [43] [DEBUG] POST /content/missing kafka_1 | [2020-10-26 15:05:39,388] INFO [GroupMetadataManager brokerId=1001] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:25 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:33 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:06:51 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:06:52 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:04 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:05 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:17 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:18 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:24 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:07:43 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:07:45 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:03 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:05 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:06 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:15 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:16 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:27 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:40 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:08:47 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:08:49 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:06 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:08 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:18 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:19 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:29 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:30 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:34 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:39 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:41 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:09:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:09:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:23 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:27 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:28 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:32 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:36 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:38 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:49 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:51 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:10:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:10:59 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:34 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:36 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:11:54 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:11:55 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:14 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:15 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:30 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:31 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:45 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:46 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:12:57 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:12:58 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:12 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | ERROR:root:An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | [2020-10-26 15:13:13,789] ERROR in app: Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:swh.storage.api.server:Exception on /content/add [POST] swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-storage_1 | ERROR:root:500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application. swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | ConnectionResetError: [Errno 104] Connection reset by peer swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send swh-storage_1 | timeout=timeout swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen swh-storage_1 | method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment swh-storage_1 | raise six.reraise(type(error), error, _stacktrace) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise swh-storage_1 | raise value.with_traceback(tb) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen swh-storage_1 | chunked=chunked, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request swh-storage_1 | conn.request(method, url, **httplib_request_kw) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1277, in request swh-storage_1 | self._send_request(method, url, body, headers, encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request swh-storage_1 | self.endheaders(body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders swh-storage_1 | self._send_output(message_body, encode_chunked=encode_chunked) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 1071, in _send_output swh-storage_1 | self.send(chunk) swh-storage_1 | File "/usr/local/lib/python3.7/http/client.py", line 993, in send swh-storage_1 | self.sock.sendall(data) swh-storage_1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 253, in raw_verb swh-storage_1 | return getattr(self.session, verb)(self._url(endpoint), **opts) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 578, in post swh-storage_1 | return self.request('POST', url, data=data, json=json, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 530, in request swh-storage_1 | resp = self.send(prep, **send_kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send swh-storage_1 | r = adapter.send(request, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/requests/adapters.py", line 498, in send swh-storage_1 | raise ConnectionError(err, request=request) swh-storage_1 | requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request swh-storage_1 | rv = self.dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request swh-storage_1 | return self.view_functions[rule.endpoint](**req.view_args) swh-storage_1 | File "<decorator-gen-4>", line 2, in content_add swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 147, in _negotiate swh-storage_1 | return f.negotiator(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/negotiation.py", line 81, in __call__ swh-storage_1 | result = self.func(*args, **kwargs) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 452, in _f swh-storage_1 | return obj_meth(**kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 24, in d swh-storage_1 | return f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/metrics.py", line 77, in d swh-storage_1 | r = f(*a, **kw) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/postgresql/storage.py", line 218, in content_add swh-storage_1 | objstorage_summary = self.objstorage.content_add(contents) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/objstorage.py", line 62, in content_add swh-storage_1 | summary = self.objstorage.add_batch({cont.sha1: cont.data for cont in contents}) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/objstorage/api/client.py", line 47, in add_batch swh-storage_1 | {"contents": contents, "check_presence": check_presence,}, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 271, in post swh-storage_1 | **opts, swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 255, in raw_verb swh-storage_1 | raise self.api_exception(e) swh-storage_1 | swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) swh-storage_1 | swh-storage_1 | During handling of the above exception, another exception occurred: swh-storage_1 | swh-storage_1 | Traceback (most recent call last): swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app swh-storage_1 | response = self.full_dispatch_request() swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request swh-storage_1 | rv = self.handle_user_exception(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1822, in handle_user_exception swh-storage_1 | return handler(e) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 47, in my_error_handler swh-storage_1 | return error_handler(exception, encode_data) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 398, in error_handler swh-storage_1 | response = encoder(exception_to_dict(exception)) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 374, in encode_data_server swh-storage_1 | encoded_data = ENCODERS[content_type](data, extra_encoders=extra_type_encoders) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/serializers.py", line 226, in msgpack_dumps swh-storage_1 | return msgpack.packb(data, use_bin_type=True, default=encode_types) swh-storage_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb swh-storage_1 | return Packer(**kwargs).pack(o) swh-storage_1 | File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 258, in msgpack._cmsgpack.Packer._pack swh-storage_1 | File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack swh-storage_1 | TypeError: can not serialize 'ConnectionError' object swh-loader_1 | [2020-10-26 15:13:13,841: ERROR/ForkPoolWorker-1] Loading failure, updating to `partial` status swh-loader_1 | Traceback (most recent call last): swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 319, in load swh-loader_1 | self.store_data() swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 418, in store_data swh-loader_1 | self.storage.content_add([obj]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 97, in content_add swh-loader_1 | return self.flush(["content"]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/buffer.py", line 137, in flush swh-loader_1 | stats = add_fn(list(batch)) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/filter.py", line 43, in content_add swh-loader_1 | [x for x in content if x.sha256 in contents_to_add] swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add swh-loader_1 | return self.post("content/add", {"content": content}) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 277, in post swh-loader_1 | return self._decode_response(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 351, in _decode_response swh-loader_1 | self.raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status swh-loader_1 | super().raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 341, in raise_for_status swh-loader_1 | raise exception from None swh-loader_1 | swh.core.api.RemoteException: <RemoteException 500 InternalServerError: []> swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /origin/visit_status/add swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /flush swh-storage_1 | [2020-10-26 15:13:13 +0000] [43] [DEBUG] POST /content/missing swh-storage_1 | [2020-10-26 15:13:14 +0000] [43] [DEBUG] POST /content/add swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] POST /clear/buffer swh-storage_1 | [2020-10-26 15:13:23 +0000] [43] [DEBUG] Closing connection.