Page MenuHomeSoftware Heritage
Create Task
Maniphest T3190

counters: Error during directory topic ingestion
Closed, MigratedEdits Locked
Actions

Description

Several error are logged during the directory topic ingestion :

Apr 01 07:36:41 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:43 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:43 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:46 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:46 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:47 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:48 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:50 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 4526518 of 9496 bytes failed: Local: Invalid compressed data"}
Apr 01 07:36:51 counters1 swh[437167]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_BAD_COMPRESSION,val=-198,str="Decompression (codec 0x4) of message at 5958409 of 8361 bytes failed: Local: Invalid compressed data"}

the good news is the position is not commited when the error is raised so it will not be necessary to restart from the beginning.

Revisions and Commits

rSPSITE puppet-swh-site
D5399rSPSITE2c2e7ed2403f counters: allow to consume big messages of the directory topic

Related Objects
Search...

Event Timeline

vsellier changed the task status from Open to Work in Progress.Apr 1 2021, 9:38 AM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier updated the task description. (Show Details)Apr 1 2021, 9:46 AM
vsellier added a comment.Apr 1 2021, 10:03 AM

The same problem occured during the poc, theses messages were ignored by using this consumer configuration "errors.tolerance": 'all' [1].
I will try to find if there is a more elegant way to deal with this issue ;)

[1] https://forge.softwareheritage.org/source/snippets/browse/master/vsellier/counters/counter.py$127

vsellier added a comment.EditedApr 1 2021, 10:32 AM

for the record, increasing the property message.max.bytes to 100 * 1024 * 1024 in the consumer configuration is not solving the problem

vlorentz added a subscriber: vlorentz.Apr 1 2021, 10:41 AM

This looks like https://github.com/edenhill/librdkafka/issues/2672 , which is caused by a bug in zstd: https://github.com/facebook/zstd/issues/2222

vlorentz added a comment.Apr 1 2021, 10:45 AM

What version of zstd is installed on the system? that bug was introduced in v1.4.5 (https://github.com/facebook/zstd/commit/718f00ff6fe42db7e6ba09a7f7992b3e85283f77), and fixed in v1.4.7 (https://github.com/facebook/zstd/pull/2272/commits/1302f8d67691356b2f0aec8c62c1a7af2886a7cc)

vsellier added a comment.EditedApr 1 2021, 11:35 AM

It seems the problem is not present anymore with a higher max message size ('500 * 1024 * 1024').

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

https://archive.softwareheritage.org/swh:1:cnt:e5dc5473a33bfea620c90f161eb684437687cd03;origin=https://forge.softwareheritage.org/source/swh-journal.git;visit=swh:1:snp:8fb83b9f652cca61ae8fe920618bb2ebb957dcf8;anchor=swh:1:rev:7815cad28aa53fac456c5fed69d5230e080e33cb;path=/swh/journal/writer/kafka.py;lines=111/

ardumont added a subscriber: ardumont.Apr 1 2021, 11:49 AM

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

Do you need such improvment though? According to the code you linked, you could pass a
producer_config dict with that key and value.

vsellier mentioned this in T3191: journal-client: Add support of max message size configuration.Apr 1 2021, 11:52 AM
vlorentz added a comment.Apr 1 2021, 11:52 AM

@ardumont the linked code is for producer, the bug is in consumers

vsellier added a comment.Apr 1 2021, 12:31 PM
In T3190#61917, @ardumont wrote:

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

Do you need such improvment though? According to the code you linked, you could pass a
producer_config dict with that key and value.

I took the code from the producer implementation as an example, but I missed the consumer already supports any configuration property via kwargs :)
So finally, I just have to add the property in the journal section in the configuration file.

vsellier added a revision: D5399: counters: allow to consume big messages of the directory topic.Apr 1 2021, 12:37 PM
vsellier added a commit: rSPSITE2c2e7ed2403f: counters: allow to consume big messages of the directory topic.Apr 1 2021, 12:54 PM
vsellier closed this task as Resolved.Apr 1 2021, 2:18 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:02 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal