Page MenuHomeSoftware Heritage

staging - counters journal client failed to process origins
Closed, ResolvedPublic

Description

Oct 14 16:25:43 counters0 systemd[1]: Started Software Heritage Counters Journal Client.
Oct 14 16:25:47 counters0 swh[1174459]: Traceback (most recent call last):
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/bin/swh", line 11, in <module>
Oct 14 16:25:47 counters0 swh[1174459]:     load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
Oct 14 16:25:47 counters0 swh[1174459]:     return swh(auto_envvar_prefix="SWH")
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
Oct 14 16:25:47 counters0 swh[1174459]:     return self.main(*args, **kwargs)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
Oct 14 16:25:47 counters0 swh[1174459]:     rv = self.invoke(ctx)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Oct 14 16:25:47 counters0 swh[1174459]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Oct 14 16:25:47 counters0 swh[1174459]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
Oct 14 16:25:47 counters0 swh[1174459]:     return ctx.invoke(self.callback, **ctx.params)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
Oct 14 16:25:47 counters0 swh[1174459]:     return callback(*args, **kwargs)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
Oct 14 16:25:47 counters0 swh[1174459]:     return f(get_current_context(), *args, **kwargs)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/counters/cli.py", line 80, in journal_client
Oct 14 16:25:47 counters0 swh[1174459]:     nb_messages = client.process(worker_fn)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 265, in process
Oct 14 16:25:47 counters0 swh[1174459]:     batch_processed, at_eof = self.handle_messages(messages, worker_fn)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/counters/kafka_client.py", line 42, in handle_messages
Oct 14 16:25:47 counters0 swh[1174459]:     worker_fn(dict(objects))
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/counters/journal_client.py", line 26, in process_journal_messages
Oct 14 16:25:47 counters0 swh[1174459]:     process_origins(messages["origin"], counters)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/counters/journal_client.py", line 49, in process_origins
Oct 14 16:25:47 counters0 swh[1174459]:     counters.add(k, v)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
Oct 14 16:25:47 counters0 swh[1174459]:     return self.post(meth._endpoint_path, post_data)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 262, in post
Oct 14 16:25:47 counters0 swh[1174459]:     data = self._encode_data(data)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 281, in _encode_data
Oct 14 16:25:47 counters0 swh[1174459]:     return encode_data(data, extra_encoders=self.extra_type_encoders)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/api/serializers.py", line 126, in encode_data_client
Oct 14 16:25:47 counters0 swh[1174459]:     return msgpack_dumps(data, extra_encoders=extra_encoders)
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/swh/core/api/serializers.py", line 275, in msgpack_dumps
Oct 14 16:25:47 counters0 swh[1174459]:     default=encode_types,
Oct 14 16:25:47 counters0 swh[1174459]:   File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 35, in packb
Oct 14 16:25:47 counters0 swh[1174459]:     return Packer(**kwargs).pack(o)
Oct 14 16:25:47 counters0 swh[1174459]:   File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
Oct 14 16:25:47 counters0 swh[1174459]:   File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
Oct 14 16:25:47 counters0 swh[1174459]:   File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
Oct 14 16:25:47 counters0 swh[1174459]:   File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
Oct 14 16:25:47 counters0 swh[1174459]:   File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
Oct 14 16:25:47 counters0 swh[1174459]: TypeError: can not serialize 'set' object
Oct 14 16:25:47 counters0 systemd[1]: swh-counters-journal-client.service: Main process exited, code=exited, status=1/FAILURE

All the packages are up-to-date

Event Timeline

vsellier renamed this task from staging - counter journal client failed to process origins to staging - counters journal client failed to process origins.Oct 14 2021, 6:30 PM
vsellier created this task.
vsellier changed the task status from Open to Work in Progress.Oct 15 2021, 9:46 AM
vsellier claimed this task.

This is the content of the netloc causing the exception:

Oct 15 07:45:21 counters0 swh[1254115]: INFO:swh.counters.journal_client:origin_netloc:bitbucket.org : {'https://bitbucket.org/trackerlab/boxes-public.git', 'https://bitbucket.org/fet_lab/neurotables.git', 'https://bitbucket.org/DataMinerUK/infinite-interns-2.git'}

@anlambert according to T3402#67318 the counters are not used to count the origins per forge. Is this portion of code still needed ?

rDCNTcd595e71aef44f93972013fb13bf1f77312283ea

Fixed by D6481
Version swh-counters:v0.9.0 released and deployed in staging and production