Page MenuHomeSoftware Heritage
Paste P204

listener failure on running out of disk in prado (db host vm)
ActivePublic

Authored by ardumont on Dec 15 2017, 9:37 AM.
-- Unit swh-scheduler-listener.service has finished starting up.
--
-- The start-up result is done.
Dec 15 01:04:47 saatchi python3[26433]: Traceback (most recent call last):
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
Dec 15 01:04:47 saatchi python3[26433]: "__main__", mod_spec)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
Dec 15 01:04:47 saatchi python3[26433]: exec(code, run_globals)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 156, in <module>
Dec 15 01:04:47 saatchi python3[26433]: event_monitor(main_app, main_backend)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 151, in event_monitor
Dec 15 01:04:47 saatchi python3[26433]: recv.capture(limit=None, timeout=None, wakeup=True)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/celery/events/__init__.py", line 339, in capture
Dec 15 01:04:47 saatchi python3[26433]: return list(self.consume(limit=limit, timeout=timeout, wakeup=wakeup))
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/mixins.py", line 199, in consume
Dec 15 01:04:47 saatchi python3[26433]: conn.drain_events(timeout=safety_interval)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/connection.py", line 275, in drain_events
Dec 15 01:04:47 saatchi python3[26433]: return self.transport.drain_events(self.connection, **kwargs)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events
Dec 15 01:04:47 saatchi python3[26433]: return connection.drain_events(**kwargs)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/amqp/connection.py", line 326, in drain_events
Dec 15 01:04:47 saatchi python3[26433]: return amqp_method(channel, args, content)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/amqp/channel.py", line 1906, in _basic_deliver
Dec 15 01:04:47 saatchi python3[26433]: fun(msg)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/messaging.py", line 598, in _receive_callback
Dec 15 01:04:47 saatchi python3[26433]: return on_m(message) if on_m else self.receive(decoded, message)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/messaging.py", line 564, in receive
Dec 15 01:04:47 saatchi python3[26433]: [callback(body, message) for callback in callbacks]
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/kombu/messaging.py", line 564, in <listcomp>
Dec 15 01:04:47 saatchi python3[26433]: [callback(body, message) for callback in callbacks]
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 38, in _receive
Dec 15 01:04:47 saatchi python3[26433]: self.process(type, body, message)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 44, in process
Dec 15 01:04:47 saatchi python3[26433]: handler and handler(event, message)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 136, in task_failed
Dec 15 01:04:47 saatchi python3[26433]: 'message': message,
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 88, in queue_action
Dec 15 01:04:47 saatchi python3[26433]: try_perform_actions()
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 62, in try_perform_actions
Dec 15 01:04:47 saatchi python3[26433]: perform_actions(actions)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/celery_backend/listener.py", line 78, in perform_actions
Dec 15 01:04:47 saatchi python3[26433]: function(*args, **kwargs)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/backend.py", line 37, in wrapped
Dec 15 01:04:47 saatchi python3[26433]: ret = fn(self, *args, **kwargs)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/swh/scheduler/backend.py", line 432, in end_task_run
Dec 15 01:04:47 saatchi python3[26433]: (backend_id, status, metadata, timestamp)
Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 223, in execute
Dec 15 01:04:47 saatchi python3[26433]: return super(RealDictCursor, self).execute(query, vars)
Dec 15 01:04:47 saatchi python3[26433]: psycopg2.OperationalError: could not extend file "base/16408/1527473.39": No space left on device
Dec 15 01:04:47 saatchi python3[26433]: HINT: Check free disk space.
Dec 15 01:04:47 saatchi python3[26433]: CONTEXT: SQL function "swh_scheduler_end_task_run" statement 1
Dec 15 01:04:47 saatchi systemd[1]: swh-scheduler-listener.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 01:04:47 saatchi systemd[1]: swh-scheduler-listener.service: Unit entered failed state.
Dec 15 01:04:47 saatchi systemd[1]: swh-scheduler-listener.service: Failed with result 'exit-code'.
Dec 15 01:04:47 saatchi systemd[1]: swh-scheduler-listener.service: Service hold-off time over, scheduling restart.
Dec 15 01:04:47 saatchi systemd[1]: Stopped Software Heritage scheduler listener.
-- Subject: Unit swh-scheduler-listener.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit swh-scheduler-listener.service has finished shutting down.
Dec 15 01:04:47 saatchi systemd[1]: swh-scheduler-listener.service: Start request repeated too quickly.
Dec 15 01:04:47 saatchi systemd[1]: Failed to start Software Heritage scheduler listener.
-- Subject: Unit swh-scheduler-listener.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--