Page MenuHomeSoftware Heritage

storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed
Closed, ResolvedPublic

Description

All _add entrypoints raise that issue at some point or another (not limited to but see [1] [2] [3] for example).

We need to find a way to circumvent this (that's noisy).

Either:

  • trap this exception and raise a wrapped swh one
  • stop using pickle as serialization?
  • other ideas?

[1] https://sentry.softwareheritage.org/share/issue/2a7f2df8dac3494fae5da930e27e4774/

[2] https://sentry.softwareheritage.org/share/issue/9bc7f0a095194c07b2b5a9f364c08cea/

[3] https://sentry.softwareheritage.org/share/issue/11e3f5a116134f469751321aab682775/

Event Timeline

ardumont triaged this task as Normal priority.Jan 21 2020, 11:36 AM
ardumont created this task.
ardumont added a project: Storage manager.
ardumont updated the task description. (Show Details)

Relatedly, errors raised by tenacity cannot be pickled because they contain a Lock: https://github.com/jd/tenacity/issues/147

vlorentz claimed this task.Feb 5 2020, 4:18 PM

python-cassandra too has some un-unpicklable errors:

Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/home/vlorentz/swh-journal/swh/journal/replay.py", line 240, in _insert_objects
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     storage.content_add_metadata([object_])
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/home/vlorentz/.local/lib/python3.7/site-packages/swh.core-0.0.86-py3.7.egg/swh/core/api/__init__.py", line 148, in meth_
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     return self.post(meth._endpoint_path, post_data)
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/home/vlorentz/.local/lib/python3.7/site-packages/swh.core-0.0.86-py3.7.egg/swh/core/api/__init__.py", line 220, in post
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     return self._decode_response(response)
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/home/vlorentz/.local/lib/python3.7/site-packages/swh.core-0.0.86-py3.7.egg/swh/core/api/__init__.py", line 283, in _decode_response
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     self.raise_for_status(response)
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/home/vlorentz/.local/lib/python3.7/site-packages/swh.core-0.0.86-py3.7.egg/swh/core/api/__init__.py", line 258, in raise_for_status
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     exception = pickle.loads(data)
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/usr/lib/python3/dist-packages/cassandra/__init__.py", line 472, in __init__
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     Timeout.__init__(self, message, **kwargs)
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:   File "/usr/lib/python3/dist-packages/cassandra/__init__.py", line 423, in __init__
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]:     kwargs["write_type"] = WriteType.value_to_name[kwargs["write_type"]]
Feb 17 11:11:59 cassandra-replay01 replayer-12305[49281]: KeyError: None
olasd added a subscriber: olasd.Feb 25 2020, 11:16 AM

@vlorentz I guess this could be closed?

ardumont closed this task as Resolved.Feb 26 2020, 10:48 AM

I think so (latest storage version deployed).

Plus, checking the last occurences of those errors referenced in sentry, the most recent date back from 18/02/2020).

Closing.

And thanks ;)