Page MenuHomeSoftware Heritage

journal publisher may crash on HTTP timeouts
Closed, MigratedEdits Locked

Description

On my docker env, I often see the journal publisher die because of:

swh-journal-publisher_1       | Traceback (most recent call last):
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
swh-journal-publisher_1       |     "__main__", mod_spec)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
swh-journal-publisher_1       |     exec(code, run_globals)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 239, in <module>
swh-journal-publisher_1       |     main()
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
swh-journal-publisher_1       |     return self.main(*args, **kwargs)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
swh-journal-publisher_1       |     rv = self.invoke(ctx)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
swh-journal-publisher_1       |     return ctx.invoke(self.callback, **ctx.params)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
swh-journal-publisher_1       |     return callback(*args, **kwargs)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 237, in main
swh-journal-publisher_1       |     publisher.poll()
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 121, in poll
swh-journal-publisher_1       |     new_objects = self.process_objects(messages)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 151, in process_objects
swh-journal-publisher_1       |     for key, value in messages.items()
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 151, in <dictcomp>
swh-journal-publisher_1       |     for key, value in messages.items()
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/journal/publisher.py", line 176, in process_contents
swh-journal-publisher_1       |     (c[b'sha1'] for c in content_objs))
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/storage/api/client.py", line 40, in content_get_metadata
swh-journal-publisher_1       |     return self.post('content/metadata', {'content': content})
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/core/api/__init__.py", line 185, in post
swh-journal-publisher_1       |     return self._decode_response(response)
swh-journal-publisher_1       |   File "/usr/local/lib/python3.6/site-packages/swh/core/api/__init__.py", line 220, in _decode_response
swh-journal-publisher_1       |     raise pickle.loads(decode_response(response))
swh-journal-publisher_1       | psycopg2.extensions.QueryCanceledError: canceling statement due to statement timeout

Event Timeline

douardda created this task.
vlorentz claimed this task.
vlorentz added a subscriber: vlorentz.

Closing because we removed the journal publisher. (My first WONTFIX <3)