Page MenuHomeSoftware Heritage

Investigate strange behavior seen in the rabbitmq admin web page
Closed, ResolvedPublic

Description

page: http://saatchi.internal.softwareheritage.org:15672/#/queues

Remove filters on queue (if any).
filter on queue per ready message desc
See that there are plenty of queues with uuid names with quite some unacked messages.
At least one has almost 1Gb of message as of now (a213366e-0021-34d9-87f6-e1a57b03b8dd).

The main queue of the listener is consumed from.
So that might be nothing and simply stopping the subscription to that queue somehow would be enough.
I'm not sure.

Event Timeline

ardumont created this task.Feb 24 2019, 8:09 PM
ardumont triaged this task as High priority.
douardda added a subscriber: douardda.EditedFeb 25 2019, 1:25 PM

I think these queues are result queues, since we have the result backend activated by default now, and its default value is rpc:// i.e. also amqp.

vlorentz added a subscriber: vlorentz.EditedFeb 25 2019, 1:34 PM

Yes, they seem to be results. Example payloads:

{b'traceback': None, b'result': {b'hostname': b'celery@loader_pypi.worker01', b'pid': 15046}, b'status': b'STARTED', b'task_id': b'aba57bc8-c99e-4b49-8343-9b6dba053af1', b'children': []}
{b'traceback': None, b'result': {b'status': b'uneventful'}, b'status': b'SUCCESS', b'task_id': b'aba57bc8-c99e-4b49-8343-9b6dba053af1', b'children': []}
{b'traceback': None, b'result': {b'hostname': b'celery@indexer_origin_intrinsic_metadata.worker02', b'pid': 96059}, b'status': b'STARTED', b'task_id': b'0e681b25-7991-4b04-a61b-f650c16b64e9', b'children': []}