Page MenuHomeSoftware Heritage

Some messages in the (azure) kafka cluster are too large for rdkafka clients to be able to decompress them
Open, HighPublic

Description

The content copy to S3 uses a journal client to do its work.

This work has been stuck for a few weeks now, because the client is breaking on some messages at the head of the swh.journal.objects.content topic.

After investigating, the error messages emitted by the client look like a decompression issue. We've had such issues in the graph export process, where we had to increase the journal client buffer size substantially to allow it to decompress all messages.

However, it looks like the current message size setting for the content topic overflows the max size that rdkafka is able to allocate for its decompression buffer.

We need to investigate this further, before we perform the backfill of objects in the new kafka cluster.

Dump of the current offsets of the s3 copy consumer group for further investigation:

0 	43,808,814 	38,508,457 	5,300,357 	content-replayer-001-2415c0f8-3149-49d1-90ee-aacc96e48fee:192.168.100.101/192.168.100.101
1 	43,764,359 	38,461,588 	5,302,771 	content-replayer-002-cad3546a-fe3c-4298-ba45-9a225e49f317:192.168.100.101/192.168.100.101
2 	43,766,028 	38,462,888 	5,303,140 	content-replayer-003-9fd121b8-d4bb-46ec-9b78-f9a7f7a1d99a:192.168.100.101/192.168.100.101
3 	43,749,343 	38,451,637 	5,297,706 	content-replayer-004-37c245d0-c264-4e3a-91a6-b25fc125ac14:192.168.100.101/192.168.100.101
4 	43,778,588 	38,475,127 	5,303,461 	content-replayer-005-d0045e23-21a5-4e1d-b74c-d53912664930:192.168.100.101/192.168.100.101
5 	43,774,127 	38,478,528 	5,295,599 	content-replayer-006-9243d5b6-0981-45e3-a2f4-64c7ad4c7668:192.168.100.101/192.168.100.101
6 	43,784,620 	38,489,348 	5,295,272 	content-replayer-007-c373c9ce-5ae7-4fba-8b92-a183255f1e03:192.168.100.101/192.168.100.101
7 	43,785,259 	38,480,366 	5,304,893 	content-replayer-008-e4326b1b-75d6-42a9-9318-e9d9a1322e33:192.168.100.101/192.168.100.101
8 	43,782,736 	38,480,816 	5,301,920 	content-replayer-009-421d0635-6d40-4114-865a-79dc9bced90a:192.168.100.101/192.168.100.101
9 	43,770,757 	38,468,204 	5,302,553 	content-replayer-010-74b4cae2-635f-4070-ab08-2ca605497411:192.168.100.101/192.168.100.101
10 	43,778,211 	38,478,808 	5,299,403 	content-replayer-011-61913913-2daa-4851-bded-973c98ac1ff2:192.168.100.101/192.168.100.101
11 	43,769,834 	38,470,166 	5,299,668 	content-replayer-012-44a4e668-fe2f-4b1c-9f1b-4948416185a8:192.168.100.101/192.168.100.101
12 	43,766,071 	38,468,480 	5,297,591 	content-replayer-013-18cfb1b3-7c92-4638-ad18-49f9e253bd55:192.168.100.101/192.168.100.101
13 	43,771,359 	38,474,437 	5,296,922 	content-replayer-014-cdc2a04d-378c-43b1-a878-d6dee3d62850:192.168.100.101/192.168.100.101
14 	43,783,204 	38,477,149 	5,306,055 	content-replayer-015-7233b907-9827-4c4b-be01-325ce999a96b:192.168.100.101/192.168.100.101
15 	43,755,937 	38,458,452 	5,297,485 	content-replayer-016-5818339f-ef42-4d19-8d24-6b72a5a7dd02:192.168.100.101/192.168.100.101
16 	43,773,488 	38,468,915 	5,304,573 	content-replayer-017-821bf608-28b7-4dca-b0c2-be34a234c203:192.168.100.101/192.168.100.101
17 	43,798,030 	38,498,789 	5,299,241 	content-replayer-018-9897b14b-dbf6-465d-adeb-8c9895970983:192.168.100.101/192.168.100.101
18 	43,778,690 	33,332,557 	10,446,133 	content-replayer-019-c9c45e6d-b914-4452-b83f-cdd31ddc7379:192.168.100.101/192.168.100.101
19 	43,768,898 	38,466,640 	5,302,258 	content-replayer-020-c56c2116-d9c1-47be-97e3-d8cc42963aee:192.168.100.101/192.168.100.101
20 	43,772,490 	38,474,979 	5,297,511 	content-replayer-021-c432e391-79e9-4a85-a6a8-46c9c1ac4a31:192.168.100.101/192.168.100.101
21 	43,753,994 	38,452,094 	5,301,900 	content-replayer-022-3da52e78-1f75-4433-b045-8b2e3569b69e:192.168.100.101/192.168.100.101
22 	43,780,022 	38,474,986 	5,305,036 	content-replayer-023-3c794fa0-0f56-4f87-9ec8-e1385138428a:192.168.100.101/192.168.100.101
23 	43,773,399 	38,470,376 	5,303,023 	content-replayer-024-08d21842-ff32-4fd4-960e-a39958bb810a:192.168.100.101/192.168.100.101
24 	43,795,442 	38,494,594 	5,300,848 	content-replayer-025-86e1cfe9-f180-4d4d-98d5-1ea5b151d573:192.168.100.101/192.168.100.101
25 	43,786,240 	38,481,916 	5,304,324 	content-replayer-026-d72ec6e5-1c5f-4db6-a650-2d25c06c9f6a:192.168.100.101/192.168.100.101
26 	43,789,380 	38,489,797 	5,299,583 	content-replayer-027-88d20a54-6823-47db-9da1-266c84c8c0dc:192.168.100.101/192.168.100.101
27 	43,788,829 	38,484,875 	5,303,954 	content-replayer-028-047ab787-4fe2-4f86-89d5-9c54c12397a5:192.168.100.101/192.168.100.101
28 	43,778,021 	38,478,584 	5,299,437 	content-replayer-029-7cb416f1-4ee3-400c-a904-25fd73d3142d:192.168.100.101/192.168.100.101
29 	43,771,217 	38,475,656 	5,295,561 	content-replayer-030-195e41ee-9c68-4a1a-92a7-a30cbd27b67f:192.168.100.101/192.168.100.101
30 	43,775,710 	38,475,824 	5,299,886 	content-replayer-031-29c5ce7e-4c3b-40af-a41e-b2efa50cf34f:192.168.100.101/192.168.100.101
31 	43,758,531 	38,460,661 	5,297,870 	content-replayer-032-052a6e10-38cf-43bf-a24f-38d1f9f169db:192.168.100.101/192.168.100.101
32 	43,762,530 	38,455,674 	5,306,856 	content-replayer-033-152cdcd0-a3fe-4f08-a52e-34f562f79523:192.168.100.101/192.168.100.101
33 	43,760,864 	38,457,085 	5,303,779 	content-replayer-034-3b7d62e9-dd03-4d76-82d4-d69b01ff4d4f:192.168.100.101/192.168.100.101
34 	43,778,277 	38,474,271 	5,304,006 	content-replayer-035-69c8a357-3051-4378-a619-0e25e88f98da:192.168.100.101/192.168.100.101
35 	43,772,836 	38,469,713 	5,303,123 	content-replayer-036-835b6db2-7fd5-4255-ae0f-5eb7c56b4988:192.168.100.101/192.168.100.101
36 	43,783,402 	38,483,318 	5,300,084 	content-replayer-037-d8d4c4b4-a868-4f1f-8dec-c32d6d4d9504:192.168.100.101/192.168.100.101
37 	43,781,593 	38,477,029 	5,304,564 	content-replayer-038-425d6a72-3614-4ec3-9b03-5c72f62a89e8:192.168.100.101/192.168.100.101
38 	43,775,490 	38,478,452 	5,297,038 	content-replayer-039-fe8220cf-1cfd-464d-9c8a-c33693339375:192.168.100.101/192.168.100.101
39 	43,762,516 	38,460,320 	5,302,196 	content-replayer-040-c91b786d-00f4-4cc3-bd7a-4f50afd8cac3:192.168.100.101/192.168.100.101
40 	43,778,957 	38,477,170 	5,301,787 	content-replayer-041-6be7b6fe-67e3-42cc-9e9f-b20e2bc2ffae:192.168.100.101/192.168.100.101
41 	43,791,067 	38,486,096 	5,304,971 	content-replayer-042-1e003314-9b78-4a5d-a546-1a6eb38ec9cb:192.168.100.101/192.168.100.101
42 	43,766,031 	38,465,732 	5,300,299 	content-replayer-043-11b86415-3531-48d6-8975-da9cbd75ae49:192.168.100.101/192.168.100.101
43 	43,776,144 	38,476,523 	5,299,621 	content-replayer-044-2b1a9ca7-15dd-4151-a90e-21a684052413:192.168.100.101/192.168.100.101
44 	43,770,876 	38,473,632 	5,297,244 	content-replayer-045-49fef889-57ce-48a0-99a0-16c3a47bb160:192.168.100.101/192.168.100.101
45 	43,757,011 	38,457,309 	5,299,702 	content-replayer-046-8bb5c048-e2f9-4a51-93e6-c29f703dd901:192.168.100.101/192.168.100.101
46 	43,766,930 	38,463,372 	5,303,558 	content-replayer-047-cf4af560-016f-4f59-93a0-8a4514b25b56:192.168.100.101/192.168.100.101
47 	43,786,162 	38,488,466 	5,297,696 	content-replayer-048-0a283464-fe51-4153-b93a-13d431558069:192.168.100.101/192.168.100.101
48 	43,796,390 	38,492,876 	5,303,514 	content-replayer-049-7079028a-9704-417e-b016-5ab0493b73cb:192.168.100.101/192.168.100.101
49 	43,778,919 	38,481,660 	5,297,259 	content-replayer-050-84b515be-d58d-4533-b7f5-285143b91ddf:192.168.100.101/192.168.100.101
50 	43,762,132 	38,462,065 	5,300,067 	content-replayer-051-e362d521-3c89-4479-be6c-9acdc15e9e36:192.168.100.101/192.168.100.101
51 	43,758,103 	38,459,197 	5,298,906 	content-replayer-052-c8429783-77c6-482d-aacf-d99b067f3ac4:192.168.100.101/192.168.100.101
52 	43,779,788 	38,477,287 	5,302,501 	content-replayer-053-5cc848cc-0253-4d77-8f4a-c8552df0bd99:192.168.100.101/192.168.100.101
53 	43,764,912 	38,468,235 	5,296,677 	content-replayer-054-56ae1086-d0bb-4463-9458-0bce9286b7cd:192.168.100.101/192.168.100.101
54 	43,770,778 	38,471,619 	5,299,159 	content-replayer-055-d20cb6dc-be3d-4824-b1c6-acef266159ae:192.168.100.101/192.168.100.101
55 	43,772,019 	38,471,246 	5,300,773 	content-replayer-056-2a936d9a-efac-4968-a78a-d37eeccc4d21:192.168.100.101/192.168.100.101
56 	43,754,511 	38,456,316 	5,298,195 	content-replayer-057-5508a50c-5485-4922-9d73-e7f6d70f7ac1:192.168.100.101/192.168.100.101
57 	43,763,310 	38,461,341 	5,301,969 	content-replayer-058-4f32ef85-8d4c-4288-99b0-dcfeb911d73c:192.168.100.101/192.168.100.101
58 	43,796,248 	38,494,029 	5,302,219 	content-replayer-059-710a878a-f574-4a27-9698-82d836b733a1:192.168.100.101/192.168.100.101
59 	43,769,705 	38,468,983 	5,300,722 	content-replayer-060-6d2ed7e5-a5ff-4355-872c-a6ff03d040bc:192.168.100.101/192.168.100.101
60 	43,774,695 	38,472,929 	5,301,766 	content-replayer-061-7e73ee12-975f-492b-b190-8dc227f75111:192.168.100.101/192.168.100.101
61 	43,753,497 	38,455,881 	5,297,616 	content-replayer-062-2724ce1b-4b25-417b-8ed0-7a9253706641:192.168.100.101/192.168.100.101
62 	43,762,932 	38,456,275 	5,306,657 	content-replayer-063-5368fe2d-d0c0-4577-ae16-5495d0e3e471:192.168.100.101/192.168.100.101
63 	43,740,624 	38,146,048 	5,594,576 	content-replayer-064-884a910d-b283-45fb-b5ff-f6ffad86099a:192.168.100.101/192.168.100.101
64 	43,779,366 	38,474,102 	5,305,264 	content-replayer-065-65fb5524-7f73-4300-a5c6-622a1e379008:192.168.100.101/192.168.100.101
65 	43,761,014 	38,463,531 	5,297,483 	content-replayer-066-ac8f6a14-e0f4-458f-9d68-8a63e91fcac5:192.168.100.101/192.168.100.101
66 	43,800,626 	38,498,165 	5,302,461 	content-replayer-067-b8545453-9257-4676-8f55-1b64e374b8ff:192.168.100.101/192.168.100.101
67 	43,787,515 	38,483,819 	5,303,696 	content-replayer-068-c7e9bf24-0022-4e32-b69e-6ba9581e3452:192.168.100.101/192.168.100.101
68 	43,771,044 	38,472,573 	5,298,471 	content-replayer-069-b9374949-7f1c-4df8-b585-f801a6273e48:192.168.100.101/192.168.100.101
69 	43,757,448 	38,455,690 	5,301,758 	content-replayer-070-07861521-1384-4ebb-947b-dc15c94659b7:192.168.100.101/192.168.100.101
70 	43,789,111 	38,490,441 	5,298,670 	content-replayer-071-d841e6cb-535b-4287-aa36-0c1a321ea2e8:192.168.100.101/192.168.100.101
71 	43,783,000 	38,479,549 	5,303,451 	content-replayer-072-5ff89acc-096f-45b7-8e30-68148281ef71:192.168.100.101/192.168.100.101
72 	43,739,919 	38,439,393 	5,300,526 	content-replayer-073-649becda-6971-4a54-92fc-b6157bb8c503:192.168.100.101/192.168.100.101
73 	43,779,607 	38,483,036 	5,296,571 	content-replayer-074-cd6fc315-5f60-4d2a-bdcc-3613ebfd501f:192.168.100.101/192.168.100.101
74 	43,806,272 	38,501,300 	5,304,972 	content-replayer-075-22419a81-fe75-4fd6-93d8-3f9ce45c8523:192.168.100.101/192.168.100.101
75 	43,741,622 	38,441,328 	5,300,294 	content-replayer-076-2526305a-d990-4855-96ec-6b1853301fbc:192.168.100.101/192.168.100.101
76 	43,781,295 	38,475,032 	5,306,263 	content-replayer-077-736931e3-2f0e-4cb5-8f65-3a2eaa021b38:192.168.100.101/192.168.100.101
77 	43,796,830 	38,494,109 	5,302,721 	content-replayer-078-544053cb-2cc3-47d2-99a1-6b29d9b4ba5d:192.168.100.101/192.168.100.101
78 	43,806,179 	38,505,035 	5,301,144 	content-replayer-079-3d7737f7-2a54-4ab4-8edd-ae3fe7d6e665:192.168.100.101/192.168.100.101
79 	43,779,140 	38,481,774 	5,297,366 	content-replayer-080-5b467d8e-28ab-45de-9c33-09c20e7e7ed1:192.168.100.101/192.168.100.101
80 	43,773,388 	38,461,946 	5,311,442 	content-replayer-081-38fe52c0-10c8-4849-b554-7c9cf5b8a85c:192.168.100.101/192.168.100.101
81 	43,745,945 	38,445,507 	5,300,438 	content-replayer-082-5b81e979-adf1-41f3-85ff-df01a8349b56:192.168.100.101/192.168.100.101
82 	43,770,418 	38,474,220 	5,296,198 	content-replayer-083-b04531d9-919b-4163-82a8-59da3e9b6864:192.168.100.101/192.168.100.101
83 	43,776,040 	38,472,673 	5,303,367 	content-replayer-084-be97701c-ee81-4de6-a037-fabe2c457788:192.168.100.101/192.168.100.101
84 	43,798,037 	38,496,814 	5,301,223 	content-replayer-085-b158a38b-43a2-4452-8f4d-0d49d808fedd:192.168.100.101/192.168.100.101
85 	43,751,490 	38,455,408 	5,296,082 	content-replayer-086-c2907974-1269-4ed5-bc33-299a4011d69b:192.168.100.101/192.168.100.101
86 	43,781,265 	38,480,532 	5,300,733 	content-replayer-087-6576f8f3-2b5b-45d8-9b0c-c28ca403cad6:192.168.100.101/192.168.100.101
87 	43,786,507 	38,484,885 	5,301,622 	content-replayer-088-b6ddd64a-2bc2-4f86-b87f-9828b6f1b25b:192.168.100.101/192.168.100.101
88 	43,779,666 	38,475,032 	5,304,634 	content-replayer-089-37175b8c-10b5-46d8-be60-404b804e1142:192.168.100.101/192.168.100.101
89 	43,744,607 	38,449,547 	5,295,060 	content-replayer-090-414405f8-2288-48b3-a567-1e16d27dddcb:192.168.100.101/192.168.100.101
90 	43,796,029 	38,496,076 	5,299,953 	content-replayer-091-406b76dc-094a-46ec-822e-7df0d2fc1522:192.168.100.101/192.168.100.101
91 	43,770,468 	38,467,502 	5,302,966 	content-replayer-092-cadd2515-4d26-44e5-bb04-4eb0ae92dbce:192.168.100.101/192.168.100.101
92 	43,748,039 	38,449,785 	5,298,254 	content-replayer-093-b7ce8cbf-9a3b-4d2c-847f-1d131f9c91b6:192.168.100.101/192.168.100.101
93 	43,743,286 	38,444,471 	5,298,815 	content-replayer-094-1c2d6bba-4869-45d6-a56c-b57a4491bc17:192.168.100.101/192.168.100.101
94 	43,786,614 	38,482,271 	5,304,343 	content-replayer-095-1a10c3df-3b7c-4a3a-b1b4-c95ced1ef8b7:192.168.100.101/192.168.100.101
95 	43,772,930 	38,469,412 	5,303,518 	content-replayer-096-8e77d068-3af4-47eb-b8c5-1612a359cabd:192.168.100.101/192.168.100.101
96 	43,784,284 	38,234,780 	5,549,504 	content-replayer-097-540e8924-8066-4688-8b02-9e98a51cba57:192.168.100.101/192.168.100.101
97 	43,766,666 	38,468,052 	5,298,614 	content-replayer-098-4dcaba45-ee70-4fcc-8930-b396e3ef753b:192.168.100.101/192.168.100.101
98 	43,780,009 	38,478,366 	5,301,643 	content-replayer-099-6b766c2d-5cf8-4452-81cf-65b6731c2a66:192.168.100.101/192.168.100.101
99 	43,766,414 	38,458,674 	5,307,740 	content-replayer-100-37c3d7e4-a24e-43f6-9c2a-fc6761ba647a:192.168.100.101/192.168.100.101
100 	43,773,866 	38,474,856 	5,299,010 	content-replayer-101-915877ab-c788-4d87-ae2f-770a5d22b7e1:192.168.100.101/192.168.100.101
101 	43,782,788 	38,480,446 	5,302,342 	content-replayer-102-a0c147cb-ea42-4306-b88b-a91fa5017e73:192.168.100.101/192.168.100.101
102 	43,731,781 	38,429,251 	5,302,530 	content-replayer-103-0721e260-66af-4de9-9e5d-cfd741fcd8c6:192.168.100.101/192.168.100.101
103 	43,784,080 	38,483,313 	5,300,767 	content-replayer-104-14d6b221-a72b-4a09-a409-7aba743d6d63:192.168.100.101/192.168.100.101
104 	43,775,839 	38,479,591 	5,296,248 	content-replayer-105-2e11945a-12d3-4bf8-80db-a36ccbe1cbee:192.168.100.101/192.168.100.101
105 	43,751,959 	38,456,153 	5,295,806 	content-replayer-106-d9ddc7c5-83a6-4e92-b557-dd292da934d8:192.168.100.101/192.168.100.101
106 	43,778,361 	38,476,841 	5,301,520 	content-replayer-107-f840f792-73ef-45d6-9d1a-9f6c8f86556e:192.168.100.101/192.168.100.101
107 	43,793,787 	38,492,815 	5,300,972 	content-replayer-108-600a5081-6dc1-458b-affa-0e01f8563216:192.168.100.101/192.168.100.101
108 	43,807,179 	38,508,130 	5,299,049 	content-replayer-109-c5b4f274-9b2a-4937-aac1-a333b07813b7:192.168.100.101/192.168.100.101
109 	43,776,083 	38,475,949 	5,300,134 	content-replayer-110-bba3dbf6-7d2a-42af-9519-5fc2ebb705b6:192.168.100.101/192.168.100.101
110 	43,789,661 	38,483,000 	5,306,661 	content-replayer-111-b33536c1-ee5a-47a3-93f9-c614a37987a4:192.168.100.101/192.168.100.101
111 	43,768,298 	38,467,092 	5,301,206 	content-replayer-112-c9d51275-a191-4005-85c6-789dfe79d8f0:192.168.100.101/192.168.100.101
112 	43,788,915 	38,487,413 	5,301,502 	content-replayer-113-92d47242-7687-4f75-8348-be472da3b125:192.168.100.101/192.168.100.101
113 	43,781,639 	38,254,660 	5,526,979 	content-replayer-114-96a14196-03d3-4362-ac82-f6b8e29332a6:192.168.100.101/192.168.100.101
114 	43,811,108 	38,511,905 	5,299,203 	content-replayer-115-5b29f0df-bffd-4e0e-809a-0519a874d538:192.168.100.101/192.168.100.101
115 	43,767,742 	38,469,259 	5,298,483 	content-replayer-116-266a144d-3486-4a51-a7ac-dc7dfb3e7054:192.168.100.101/192.168.100.101
116 	43,767,690 	38,459,723 	5,307,967 	content-replayer-117-59f26700-17c7-46c4-aafe-cc7775d80403:192.168.100.101/192.168.100.101
117 	43,786,291 	38,485,332 	5,300,959 	content-replayer-118-b1905c0d-1339-405a-ae76-510982a992de:192.168.100.101/192.168.100.101
118 	43,779,420 	38,479,931 	5,299,489 	content-replayer-119-287d303d-66c7-4c86-940f-1213f51570c1:192.168.100.101/192.168.100.101
119 	43,792,351 	38,489,055 	5,303,296 	content-replayer-120-37303e43-a73f-444b-a530-f0da5d1ead2b:192.168.100.101/192.168.100.101
120 	43,788,904 	38,493,324 	5,295,580 	content-replayer-121-415a1d60-59ee-46cf-b592-4228db6c6922:192.168.100.101/192.168.100.101
121 	43,764,372 	38,465,597 	5,298,775 	content-replayer-122-f98c2473-f012-4fd3-b399-b9900f6784c1:192.168.100.101/192.168.100.101
122 	43,774,027 	38,475,590 	5,298,437 	content-replayer-123-1c73d224-5afc-41f2-bf5d-f7a2b92fed81:192.168.100.101/192.168.100.101
123 	43,740,485 	38,440,625 	5,299,860 	content-replayer-124-34a28249-f359-4e72-a58a-999bd1418897:192.168.100.101/192.168.100.101
124 	43,756,674 	38,452,328 	5,304,346 	content-replayer-125-c4fa56a4-3806-4885-abe5-99d8e63ee13b:192.168.100.101/192.168.100.101
125 	43,782,351 	38,476,689 	5,305,662 	content-replayer-126-91cbaa79-99e2-45f5-a776-ef8b44bf1988:192.168.100.101/192.168.100.101
126 	43,804,386 	38,497,870 	5,306,516 	content-replayer-127-8307ba0a-f2a1-4996-aae8-4e144a5f7c70:192.168.100.101/192.168.100.101
127 	43,798,314 	38,493,701 	5,304,613 	content-replayer-128-68755487-33db-4b99-be43-95c1cc633c5e:192.168.100.101/192.168.100.101
128 	43,781,708 	38,479,021 	5,302,687 	content-replayer-129-bb5b90c4-a117-4d47-89b0-eff062648956:192.168.100.101/192.168.100.101
129 	43,758,012 	38,457,420 	5,300,592 	content-replayer-130-7c313107-27f7-4ad4-aa5c-2d90d62a6916:192.168.100.101/192.168.100.101
130 	43,773,682 	38,470,181 	5,303,501 	content-replayer-131-1e1f4716-9996-404f-9ce2-d9f94aeac3fe:192.168.100.101/192.168.100.101
131 	43,783,030 	38,482,031 	5,300,999 	content-replayer-132-3be181f8-e024-46fd-805c-4b7beb114140:192.168.100.101/192.168.100.101
132 	43,781,353 	38,481,072 	5,300,281 	content-replayer-133-5392e25e-01f0-4ca3-810b-7d5b4979827e:192.168.100.101/192.168.100.101
133 	43,762,186 	38,458,065 	5,304,121 	content-replayer-134-306eb0b0-2d6f-4857-abad-1050962a2695:192.168.100.101/192.168.100.101
134 	43,787,509 	38,481,551 	5,305,958 	content-replayer-135-82cc56fe-64c3-4bc0-ac74-b69352a8fb39:192.168.100.101/192.168.100.101
135 	43,768,141 	38,463,591 	5,304,550 	content-replayer-136-e1261697-bba5-4c96-87cc-482100a71460:192.168.100.101/192.168.100.101
136 	43,783,840 	38,479,842 	5,303,998 	content-replayer-137-24c8ac55-36fa-43d3-8626-5e3012c95516:192.168.100.101/192.168.100.101
137 	43,782,771 	38,482,079 	5,300,692 	content-replayer-138-16c5b42b-383e-478e-aee0-e3858a214dd2:192.168.100.101/192.168.100.101
138 	43,794,455 	38,498,633 	5,295,822 	content-replayer-139-3f5c42da-2076-4331-a5e5-9d5a5e01f605:192.168.100.101/192.168.100.101
139 	43,758,501 	38,460,286 	5,298,215 	content-replayer-140-67f5aa3b-6e41-46a2-bc25-14d001de3c79:192.168.100.101/192.168.100.101
140 	43,782,354 	38,481,589 	5,300,765 	content-replayer-141-2a33c360-7089-4c9f-a11b-fd39da80eaf7:192.168.100.101/192.168.100.101
141 	43,766,958 	38,465,081 	5,301,877 	content-replayer-142-554aeb95-9868-427c-a1a0-71e21cfc984d:192.168.100.101/192.168.100.101
142 	43,785,607 	38,486,377 	5,299,230 	content-replayer-143-5d34de77-3f9c-426d-9cb8-8ff89b4ae87f:192.168.100.101/192.168.100.101
143 	43,770,060 	38,468,902 	5,301,158 	content-replayer-144-466a2415-7bd3-47b3-9449-87dc637463b9:192.168.100.101/192.168.100.101
144 	43,784,543 	38,483,489 	5,301,054 	content-replayer-145-5638ce60-5634-483d-8d2d-55b97de05d3a:192.168.100.101/192.168.100.101
145 	43,785,913 	38,480,562 	5,305,351 	content-replayer-146-36b7e0a3-e56d-46a6-bbc2-91a45c54993a:192.168.100.101/192.168.100.101
146 	43,765,864 	38,466,031 	5,299,833 	content-replayer-147-f8ee80f3-ad34-4c4a-80ee-e8ab479c5f82:192.168.100.101/192.168.100.101
147 	43,771,280 	38,460,828 	5,310,452 	content-replayer-148-8785c4a3-277c-4dd0-9188-7a2f4bd62f4d:192.168.100.101/192.168.100.101
148 	43,783,406 	38,475,307 	5,308,099 	content-replayer-149-2ce1b2b5-d9dc-4418-b227-5fa5185dc4b0:192.168.100.101/192.168.100.101
149 	43,760,654 	38,453,722 	5,306,932 	content-replayer-150-d9f8d088-efab-4321-ac40-fa71a9304f8f:192.168.100.101/192.168.100.101
150 	43,791,068 	38,487,239 	5,303,829 	content-replayer-151-cffc47c7-99ef-4f63-a981-6aa2eb9a44c7:192.168.100.101/192.168.100.101
151 	43,781,763 	38,481,834 	5,299,929 	content-replayer-152-cfd24649-0c2b-420d-a9ab-bb3b7526649d:192.168.100.101/192.168.100.101
152 	43,763,520 	38,465,432 	5,298,088 	content-replayer-153-55e5ef9b-dfef-4ca5-b043-ba1c13f1cc9e:192.168.100.101/192.168.100.101
153 	43,772,096 	38,475,014 	5,297,082 	content-replayer-154-c9f80e53-dc9b-4370-87a7-5a6c1ede14b8:192.168.100.101/192.168.100.101
154 	43,810,895 	38,510,104 	5,300,791 	content-replayer-155-39c15af5-9287-4d79-85e8-3791516726c9:192.168.100.101/192.168.100.101
155 	43,800,162 	38,494,108 	5,306,054 	content-replayer-156-241d3f8c-f3bf-452a-a53f-4ebe9b2a711c:192.168.100.101/192.168.100.101
156 	43,794,969 	38,493,505 	5,301,464 	content-replayer-157-836c61c2-cc16-4a33-a1b1-e4ea5a1a2703:192.168.100.101/192.168.100.101
157 	43,782,044 	38,476,315 	5,305,729 	content-replayer-158-e4ae7bea-4b2c-4e63-8150-adae5fa5bd6b:192.168.100.101/192.168.100.101
158 	43,762,846 	38,462,533 	5,300,313 	content-replayer-159-0dc726e5-f7f7-43f7-b72b-0f26d0c5478b:192.168.100.101/192.168.100.101
159 	43,742,298 	38,439,503 	5,302,795 	content-replayer-160-41a59db1-f474-4eb5-90bd-b567ae84fba1:192.168.100.101/192.168.100.101
160 	43,760,706 	38,456,266 	5,304,440 	content-replayer-161-8795ec89-af1a-45e7-8fd4-8f3248a0d62f:192.168.100.101/192.168.100.101
161 	43,759,677 	38,456,420 	5,303,257 	content-replayer-162-948d2e57-d100-4244-a144-e8ecc8ce1322:192.168.100.101/192.168.100.101
162 	43,760,194 	38,455,669 	5,304,525 	content-replayer-163-2e8cc10e-773b-4277-9e6a-98e32fd530a5:192.168.100.101/192.168.100.101
163 	43,774,266 	38,477,420 	5,296,846 	content-replayer-164-c2928e2b-44db-46fd-b1c5-25a063d26670:192.168.100.101/192.168.100.101
164 	43,765,863 	38,470,774 	5,295,089 	content-replayer-165-2ca05b45-580b-4050-be04-3a7fda7fa935:192.168.100.101/192.168.100.101
165 	43,765,780 	38,459,168 	5,306,612 	content-replayer-166-56d28200-5f6c-4bda-aedd-e79575335b36:192.168.100.101/192.168.100.101
166 	43,781,804 	38,484,686 	5,297,118 	content-replayer-167-1584032c-2566-4f2c-b589-738ab0065212:192.168.100.101/192.168.100.101
167 	43,790,021 	38,479,235 	5,310,786 	content-replayer-168-3a4dd029-630d-4e13-a138-2e448ae5a65b:192.168.100.101/192.168.100.101
168 	43,789,415 	38,488,545 	5,300,870 	content-replayer-169-dd8b786d-d8c8-4ca9-8f9e-d074b6b0baf7:192.168.100.101/192.168.100.101
169 	43,781,918 	38,485,573 	5,296,345 	content-replayer-170-bc78aeb4-9eb7-485b-8564-ffb9d52db56b:192.168.100.101/192.168.100.101
170 	43,763,522 	38,457,943 	5,305,579 	content-replayer-171-c1d85ee4-5186-4231-9c41-ff6060d97c4d:192.168.100.101/192.168.100.101
171 	43,755,858 	38,453,242 	5,302,616 	content-replayer-172-4fea9050-9144-4e22-ab46-2977564b213a:192.168.100.101/192.168.100.101
172 	43,803,295 	38,496,135 	5,307,160 	content-replayer-173-9e135df3-c45b-47e8-85f2-6c76af48bbb6:192.168.100.101/192.168.100.101
173 	43,790,704 	38,488,255 	5,302,449 	content-replayer-174-919d627a-f2c9-4768-bb3f-ffc6ccdf2c86:192.168.100.101/192.168.100.101
174 	43,794,028 	38,492,262 	5,301,766 	content-replayer-175-14b3f1c8-56c8-460d-b3db-efeb7e24e292:192.168.100.101/192.168.100.101
175 	43,800,849 	38,499,366 	5,301,483 	content-replayer-176-2631f913-2abd-49b1-9d9f-a0196787465f:192.168.100.101/192.168.100.101
176 	43,766,173 	38,463,245 	5,302,928 	content-replayer-177-b082e319-d03d-4a31-9d1c-e1d7ff47663e:192.168.100.101/192.168.100.101
177 	43,769,576 	38,463,744 	5,305,832 	content-replayer-178-ce104385-3522-4b6d-99d5-e5aa2564c9d4:192.168.100.101/192.168.100.101
178 	43,772,571 	38,470,594 	5,301,977 	content-replayer-179-799bb83d-3e77-4dd8-bf29-f3d2d2e4d8ba:192.168.100.101/192.168.100.101
179 	43,781,680 	38,476,212 	5,305,468 	content-replayer-180-d44886c2-d73e-4585-b4ff-afa6bf6a9c0e:192.168.100.101/192.168.100.101
180 	43,797,807 	38,495,875 	5,301,932 	content-replayer-181-2306e083-aabe-44d1-a00e-7c97d938e6c2:192.168.100.101/192.168.100.101
181 	43,767,369 	38,466,932 	5,300,437 	content-replayer-182-dd8b14e0-b545-4686-9a7a-325d2026d14a:192.168.100.101/192.168.100.101
182 	43,777,677 	38,473,582 	5,304,095 	content-replayer-183-3b83b6b1-0604-4ceb-b6ed-3fab3323ed18:192.168.100.101/192.168.100.101
183 	43,768,211 	38,469,031 	5,299,180 	content-replayer-184-5ecf84df-0358-4906-aea4-fb357155e27a:192.168.100.101/192.168.100.101
184 	43,772,895 	38,470,097 	5,302,798 	content-replayer-185-71bd1373-2b41-458f-9be1-fd60d85a23ce:192.168.100.101/192.168.100.101
185 	43,797,720 	38,491,226 	5,306,494 	content-replayer-186-fe7881e2-95b4-4220-9ac4-9e05c7329528:192.168.100.101/192.168.100.101
186 	43,794,874 	38,495,258 	5,299,616 	content-replayer-187-ab883284-fe83-4cdf-a2e1-a345c90e5775:192.168.100.101/192.168.100.101
187 	43,786,711 	38,481,069 	5,305,642 	content-replayer-188-14284ae8-347c-4596-a5bc-b93c5349f9e0:192.168.100.101/192.168.100.101
188 	43,759,119 	38,460,579 	5,298,540 	content-replayer-189-724d4e37-f749-47ef-9957-31a68fe6e646:192.168.100.101/192.168.100.101
189 	43,756,197 	38,449,002 	5,307,195 	content-replayer-190-c37a29df-443e-4fb5-8088-60a021cff072:192.168.100.101/192.168.100.101
190 	43,753,417 	38,445,819 	5,307,598 	content-replayer-191-adf92845-d37f-4644-97b8-d913ff0f0d0c:192.168.100.101/192.168.100.101
191 	43,760,327 	38,459,770 	5,300,557 	content-replayer-192-a142e409-402e-4d41-a01f-5caa5cc483f7:192.168.100.101/192.168.100.101
192 	43,774,430 	38,471,070 	5,303,360 	content-replayer-193-d674f413-a9e8-4ead-9b5a-36d94fe67172:192.168.100.101/192.168.100.101
193 	43,757,239 	38,459,581 	5,297,658 	content-replayer-194-5c67e292-45dd-4cd8-883f-62190a76e15c:192.168.100.101/192.168.100.101
194 	43,763,181 	38,461,893 	5,301,288 	content-replayer-195-3088302b-1a2d-49b7-b38b-0be45d929f77:192.168.100.101/192.168.100.101
195 	43,754,529 	38,451,515 	5,303,014 	content-replayer-196-a5576466-9f69-4791-bd30-378c2d8495ae:192.168.100.101/192.168.100.101
196 	43,796,587 	38,497,136 	5,299,451 	content-replayer-197-c53ce3fc-4c4d-4ada-bf73-7eac59be878f:192.168.100.101/192.168.100.101
197 	43,775,972 	38,478,282 	5,297,690 	content-replayer-198-da7a120c-eee7-4f7c-99fa-f37531647be3:192.168.100.101/192.168.100.101
198 	43,814,410 	38,514,237 	5,300,173 	content-replayer-199-0598e961-02f0-41e0-a33a-d158d634592b:192.168.100.101/192.168.100.101
199 	43,775,484 	38,470,406 	5,305,078 	content-replayer-200-e9f1efaa-9ae0-4be3-a055-7c7fc7c6a197:192.168.100.101/192.168.100.101
200 	43,762,904 	38,464,437 	5,298,467 	content-replayer-201-c469a355-3a26-4c67-ad1c-bbef2c5ec4ff:192.168.100.101/192.168.100.101
201 	43,774,292 	38,470,269 	5,304,023 	content-replayer-202-6943368e-3964-4205-9f77-f3535103db22:192.168.100.101/192.168.100.101
202 	43,786,147 	38,485,163 	5,300,984 	content-replayer-203-db066a94-4b2c-44df-bf58-4c269368ffbb:192.168.100.101/192.168.100.101
203 	43,773,372 	38,473,404 	5,299,968 	content-replayer-204-50ad512a-aed3-4778-a506-2c7d17b3a487:192.168.100.101/192.168.100.101
204 	43,794,669 	38,493,167 	5,301,502 	content-replayer-205-007367a7-cec6-44e4-8268-83d96dee9016:192.168.100.101/192.168.100.101
205 	43,791,233 	38,492,111 	5,299,122 	content-replayer-206-0d2c074f-aff6-471a-a06d-96928b67389f:192.168.100.101/192.168.100.101
206 	43,794,174 	38,490,602 	5,303,572 	content-replayer-207-7af6169a-950d-40a1-8925-481ab350d05e:192.168.100.101/192.168.100.101
207 	43,757,664 	38,458,276 	5,299,388 	content-replayer-208-8a150277-c667-4b2a-964b-3fbe3be0df12:192.168.100.101/192.168.100.101
208 	43,785,303 	38,482,822 	5,302,481 	content-replayer-209-3257663a-c0ec-4dd9-b0ac-fb9e1e3d9a8a:192.168.100.101/192.168.100.101
209 	43,785,100 	38,482,722 	5,302,378 	content-replayer-210-91d3745a-320a-44b7-b35f-773ce27dcb65:192.168.100.101/192.168.100.101
210 	43,803,382 	38,501,459 	5,301,923 	content-replayer-211-8ad8868e-9494-4003-9635-1659fb63487e:192.168.100.101/192.168.100.101
211 	43,773,646 	38,475,907 	5,297,739 	content-replayer-212-46af40e5-0537-421b-81d9-a3a74b305167:192.168.100.101/192.168.100.101
212 	43,770,546 	38,470,736 	5,299,810 	content-replayer-213-3c06f844-0bb1-4583-a706-8edbad305799:192.168.100.101/192.168.100.101
213 	43,758,849 	38,456,959 	5,301,890 	content-replayer-214-a925c053-e2d9-4147-944c-78f6852d6ec2:192.168.100.101/192.168.100.101
214 	43,772,180 	38,472,697 	5,299,483 	content-replayer-215-c0a43040-19b9-483e-879d-290991a81317:192.168.100.101/192.168.100.101
215 	43,817,458 	38,515,768 	5,301,690 	content-replayer-216-ac43b5cb-b9c8-4af7-8b12-c103e2f9727e:192.168.100.101/192.168.100.101
216 	43,800,084 	38,497,846 	5,302,238 	content-replayer-217-851259ae-16d0-4b46-8045-c355d7d5d734:192.168.100.101/192.168.100.101
217 	43,754,241 	38,453,578 	5,300,663 	content-replayer-218-10f87bbe-d77c-41e5-ba6c-826fc6e998a2:192.168.100.101/192.168.100.101
218 	43,773,650 	38,476,359 	5,297,291 	content-replayer-219-0db15257-8f98-41d5-89c4-24fcbfad56a5:192.168.100.101/192.168.100.101
219 	43,751,352 	38,454,594 	5,296,758 	content-replayer-220-b66cbddc-5e49-4241-885a-f5b117a27d80:192.168.100.101/192.168.100.101
220 	43,770,930 	38,470,926 	5,300,004 	content-replayer-221-02801534-388b-4a0d-b8b2-ec508dec85ad:192.168.100.101/192.168.100.101
221 	43,781,613 	38,482,391 	5,299,222 	content-replayer-222-4f49c0dd-e500-4cae-af85-259c9428b517:192.168.100.101/192.168.100.101
222 	43,744,978 	38,446,740 	5,298,238 	content-replayer-223-6a5b6445-ade3-46ee-92bd-c1a61feb28fa:192.168.100.101/192.168.100.101
223 	43,772,645 	38,463,506 	5,309,139 	content-replayer-224-6a52822b-9f88-4ad8-b982-090dfad7c9e3:192.168.100.101/192.168.100.101
224 	43,768,362 	38,467,544 	5,300,818 	content-replayer-225-32a7c833-ecc9-437c-8f29-390efeff11ef:192.168.100.101/192.168.100.101
225 	43,764,895 	38,459,600 	5,305,295 	content-replayer-226-a3a637a5-b228-4c1d-a7f1-ae8f6f94e7b7:192.168.100.101/192.168.100.101
226 	43,775,753 	38,477,037 	5,298,716 	content-replayer-227-9280b09f-bdeb-4c09-806d-21fa9f41d978:192.168.100.101/192.168.100.101
227 	43,773,196 	38,468,082 	5,305,114 	content-replayer-228-acb58472-4b39-468f-8aa0-a358aec90c68:192.168.100.101/192.168.100.101
228 	43,805,270 	38,510,358 	5,294,912 	content-replayer-229-d82aa003-78d6-48e4-9654-2906f18035ee:192.168.100.101/192.168.100.101
229 	43,775,205 	38,479,047 	5,296,158 	content-replayer-230-acd3e033-b85e-419e-b451-b90cccfc2479:192.168.100.101/192.168.100.101
230 	43,752,232 	38,453,742 	5,298,490 	content-replayer-231-9904aa8d-48f8-47f7-860d-5eff89c6aac6:192.168.100.101/192.168.100.101
231 	43,771,997 	38,470,397 	5,301,600 	content-replayer-232-7680271b-a07c-46ae-ad13-3ef10c88026a:192.168.100.101/192.168.100.101
232 	43,788,234 	38,491,614 	5,296,620 	content-replayer-233-893c9c6a-fe53-4c26-bf66-e5836ccb9392:192.168.100.101/192.168.100.101
233 	43,775,132 	38,477,167 	5,297,965 	content-replayer-234-196e3d55-4376-4dde-8f9c-1eb471759880:192.168.100.101/192.168.100.101
234 	43,795,433 	38,491,023 	5,304,410 	content-replayer-235-4424a7ea-5168-4c84-899f-0911b8960f88:192.168.100.101/192.168.100.101
235 	43,808,099 	38,508,860 	5,299,239 	content-replayer-236-9ca9d2d3-4588-44d8-8059-892854632a99:192.168.100.101/192.168.100.101
236 	43,801,129 	38,495,168 	5,305,961 	content-replayer-237-ae32ee6a-9d14-4cce-adf3-83c1fa4d11a9:192.168.100.101/192.168.100.101
237 	43,766,514 	38,464,964 	5,301,550 	content-replayer-238-e378fcaf-85e5-47c8-9a89-30adbe2844a0:192.168.100.101/192.168.100.101
238 	43,786,003 	38,485,739 	5,300,264 	content-replayer-239-3a9aa47f-66c7-4f9c-b7bf-40fb8be0b552:192.168.100.101/192.168.100.101
239 	43,775,743 	38,475,455 	5,300,288 	content-replayer-240-9e88caf2-a791-4644-9fbd-2f4d7323638b:192.168.100.101/192.168.100.101
240 	43,797,667 	38,496,325 	5,301,342 	content-replayer-241-fb79acdf-3aeb-4942-9a5c-2bf000bcec0c:192.168.100.101/192.168.100.101
241 	43,772,618 	38,471,182 	5,301,436 	content-replayer-242-9a191b33-4912-4299-90fa-a36155117629:192.168.100.101/192.168.100.101
242 	43,769,786 	38,468,056 	5,301,730 	content-replayer-243-dd1a86d0-e576-486c-b47a-696a663faddc:192.168.100.101/192.168.100.101
243 	43,768,081 	38,466,972 	5,301,109 	content-replayer-244-2ec96423-d712-410f-90aa-b50fc42d6916:192.168.100.101/192.168.100.101
244 	43,764,804 	38,462,702 	5,302,102 	content-replayer-245-8d1a8c6e-f610-4be2-ba2f-6c7a5bad6470:192.168.100.101/192.168.100.101
245 	43,817,059 	38,513,231 	5,303,828 	content-replayer-246-c1b26524-10dd-4223-b554-d395632af58c:192.168.100.101/192.168.100.101
246 	43,822,637 	38,517,625 	5,305,012 	content-replayer-247-373a224c-a74e-45e2-bd86-aacb1b89d1fc:192.168.100.101/192.168.100.101
247 	43,779,343 	38,486,693 	5,292,650 	content-replayer-248-b763225b-1014-44e2-ad31-84aa25fff33c:192.168.100.101/192.168.100.101
248 	43,768,316 	38,466,864 	5,301,452 	content-replayer-249-33d254fa-785d-4de3-886c-b7e94c12e103:192.168.100.101/192.168.100.101
249 	43,755,187 	38,449,577 	5,305,610 	content-replayer-250-9145a218-c866-42b6-86a8-9e3d74781704:192.168.100.101/192.168.100.101
250 	43,776,400 	38,247,096 	5,529,304 	content-replayer-251-a33e2c74-5b92-47d0-9cff-3d28c0b093d7:192.168.100.101/192.168.100.101
251 	43,766,793 	38,468,986 	5,297,807 	content-replayer-252-95f63adc-83d5-47b9-b700-e44443569fcd:192.168.100.101/192.168.100.101
252 	43,777,183 	38,469,717 	5,307,466 	content-replayer-253-044a9f74-5ea7-4b8c-9512-f4775fc91850:192.168.100.101/192.168.100.101
253 	43,748,887 	38,453,691 	5,295,196 	content-replayer-254-dd1f0d29-4b4c-46b2-a901-7be741e8e1d5:192.168.100.101/192.168.100.101
254 	43,760,841 	38,466,146 	5,294,695 	content-replayer-255-7077b144-4cc2-4c07-bfec-111972312c99:192.168.100.101/192.168.100.101
255 	43,783,688 	38,479,227 	5,304,461 	content-replayer-256-1e1f0f78-bcdd-44e8-b363-17f838282645:192.168.100.101/192.168.100.101

Event Timeline

olasd triaged this task as High priority.Aug 27 2020, 12:27 PM
olasd created this task.
olasd added a comment.Sep 22 2020, 6:27 PM

After resetting a local consumer to these offsets, I was completely unable to reproduce this issue.

However, restarting the "content replayers" on uffizi reproduces the issue. This smells like a bug on the rdkafka snapshot we're using in production.