Page MenuHomeSoftware Heritage

counters: Error during directory topic ingestion
Closed, ResolvedPublic

Description

Several error are logged during the directory topic ingestion :

Apr 01 07:36:41 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:43 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:43 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:46 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:46 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:47 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:48 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:50 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:51 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}

the good news is the position is not commited when the error is raised so it will not be necessary to restart from the beginning.

Event Timeline

vsellier changed the task status from Open to Work in Progress.Apr 1 2021, 9:38 AM
vsellier triaged this task as Normal priority.
vsellier created this task.

The same problem occured during the poc, theses messages were ignored by using this consumer configuration "errors.tolerance": 'all' [1].
I will try to find if there is a more elegant way to deal with this issue ;)

[1] https://forge.softwareheritage.org/source/snippets/browse/master/vsellier/counters/counter.py$127

for the record, increasing the property message.max.bytes to 100 * 1024 * 1024 in the consumer configuration is not solving the problem

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

Do you need such improvment though? According to the code you linked, you could pass a
producer_config dict with that key and value.

@ardumont the linked code is for producer, the bug is in consumers

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

Do you need such improvment though? According to the code you linked, you could pass a
producer_config dict with that key and value.

I took the code from the producer implementation as an example, but I missed the consumer already supports any configuration property via kwargs :)
So finally, I just have to add the property in the journal section in the configuration file.