Page MenuHomeSoftware Heritage
Differential D1227

[docker-compose] Ensure RabbitMQ server is up before starting related services
ClosedPublic
Actions

Authored by anlambert on Mar 7 2019, 11:49 AM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDENV19dd6fd747fb: dockerfiles: Ensure RabbitMQ server is up before starting related services
rCDFD19dd6fd747fb: dockerfiles: Ensure RabbitMQ server is up before starting related services
Summary

To ensure all RabbitMQ related services get correctly initialized, simply wait for the RabbitMQ server being up
(using the great wait-for-it script) before trying to connect to it.

For instance without that change, the scheduler-listener service fails to initialize while reporting the following errors:

swh-scheduler-listener_1      | Starting the swh-scheduler listener
swh-scheduler-listener_1      | [INFO] swh.core.config -- Loading config file /scheduler.yml
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 141, in _connect
swh-scheduler-listener_1      |     host, port, family, socket.SOCK_STREAM, SOL_TCP)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
swh-scheduler-listener_1      |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
swh-scheduler-listener_1      | socket.gaierror: [Errno -2] Name or service not known
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | During handling of the above exception, another exception occurred:
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
swh-scheduler-listener_1      |     yield
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
swh-scheduler-listener_1      |     return fun(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
swh-scheduler-listener_1      |     return self.connection
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 823, in connection
swh-scheduler-listener_1      |     self._connection = self._establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 778, in _establish_connection
swh-scheduler-listener_1      |     conn = self.transport.establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
swh-scheduler-listener_1      |     conn.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 307, in connect
swh-scheduler-listener_1      |     self.transport.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 81, in connect
swh-scheduler-listener_1      |     self._connect(self.host, self.port, self.connect_timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 152, in _connect
swh-scheduler-listener_1      |     "failed to resolve broker hostname"))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 165, in _connect
swh-scheduler-listener_1      |     self.sock.connect(sa)
swh-scheduler-listener_1      | ConnectionRefusedError: [Errno 111] Connection refused
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | During handling of the above exception, another exception occurred:
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/bin/swh-scheduler", line 11, in <module>
swh-scheduler-listener_1      |     load_entry_point('swh.scheduler', 'console_scripts', 'swh-scheduler')()
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/cli.py", line 745, in main
swh-scheduler-listener_1      |     return cli(auto_envvar_prefix='SWH_SCHEDULER')
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
swh-scheduler-listener_1      |     return self.main(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
swh-scheduler-listener_1      |     rv = self.invoke(ctx)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
swh-scheduler-listener_1      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
swh-scheduler-listener_1      |     return ctx.invoke(self.callback, **ctx.params)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
swh-scheduler-listener_1      |     return callback(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
swh-scheduler-listener_1      |     return f(get_current_context(), *args, **kwargs)
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/cli.py", line 613, in listener
swh-scheduler-listener_1      |     event_monitor(app, backend=scheduler)
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/celery_backend/listener.py", line 172, in event_monitor
swh-scheduler-listener_1      |     node_id='listener',
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/celery_backend/listener.py", line 25, in __init__
swh-scheduler-listener_1      |     channel, handlers, routing_key, node_id, app, queue_prefix, accept)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/celery/events/receiver.py", line 41, in __init__
swh-scheduler-listener_1      |     self.channel = maybe_channel(channel)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 1006, in maybe_channel
swh-scheduler-listener_1      |     return channel.default_channel
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 852, in default_channel
swh-scheduler-listener_1      |     self.ensure_connection(**conn_opts)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
swh-scheduler-listener_1      |     self.gen.throw(type, value, traceback)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
swh-scheduler-listener_1      |     sys.exc_info()[2])
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
swh-scheduler-listener_1      |     raise value.with_traceback(tb)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
swh-scheduler-listener_1      |     yield
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
swh-scheduler-listener_1      |     return fun(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
swh-scheduler-listener_1      |     return self.connection
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 823, in connection
swh-scheduler-listener_1      |     self._connection = self._establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 778, in _establish_connection
swh-scheduler-listener_1      |     conn = self.transport.establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
swh-scheduler-listener_1      |     conn.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 307, in connect
swh-scheduler-listener_1      |     self.transport.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 81, in connect
swh-scheduler-listener_1      |     self._connect(self.host, self.port, self.connect_timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 152, in _connect
swh-scheduler-listener_1      |     "failed to resolve broker hostname"))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 165, in _connect
swh-scheduler-listener_1      |     self.sock.connect(sa)
swh-scheduler-listener_1      | kombu.exceptions.OperationalError: [Errno 111] Connection refused

Diff Detail

Repository
rCDFD Dockerfiles for developers
Branch
ensure-amqp-server-up
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 4566
Build 6055: arc lint + arc unit

Event Timeline

anlambert created this revision.Mar 7 2019, 11:49 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 7 2019, 11:49 AM
Harbormaster completed remote builds in B4554: Diff 3843.Mar 7 2019, 11:49 AM
anlambert updated this revision to Diff 3845.Mar 7 2019, 3:28 PM

Update: Instead of modifying configuration, prefer an approach similar to the one
proposed in D1228 (wait for RabbitMQ server being up then start services)

Harbormaster completed remote builds in B4556: Diff 3845.Mar 7 2019, 3:28 PM
anlambert retitled this revision from [docker-compose] conf/scheduler: Increase celery max_retries value to [docker-compose] Ensure RabbitMQ server is up before starting related services.Mar 7 2019, 3:30 PM
anlambert edited the summary of this revision. (Show Details)
anlambert updated this revision to Diff 3847.Mar 7 2019, 4:46 PM

Update: Edit commit log message

Harbormaster completed remote builds in B4558: Diff 3847.Mar 7 2019, 4:46 PM
anlambert edited the summary of this revision. (Show Details)Mar 8 2019, 12:49 AM
anlambert edited the summary of this revision. (Show Details)
douardda accepted this revision.Mar 8 2019, 10:39 AM
douardda added a subscriber: douardda.

nice one, thanks!

This revision is now accepted and ready to land.Mar 8 2019, 10:39 AM
anlambert updated this revision to Diff 3854.Mar 8 2019, 11:02 AM

Rebase

Harbormaster completed remote builds in B4566: Diff 3854.Mar 8 2019, 11:02 AM
Closed by commit rCDFD19dd6fd747fb: dockerfiles: Ensure RabbitMQ server is up before starting related services (authored by anlambert). · Explain WhyMar 8 2019, 11:02 AM
This revision was automatically updated to reflect the committed changes.
zack mentioned this in D1221: readme: integrate the docker-based development setup from the main's doc.Mar 8 2019, 11:20 AM
anlambert added a commit: rDENV19dd6fd747fb: dockerfiles: Ensure RabbitMQ server is up before starting related services.Nov 26 2019, 1:35 PM

Revision Contents
Changeset List

PathSize
dockerfiles/
swh-indexer-worker/
5 lines
5 lines
swh-listers-worker/
5 lines
15 lines
swh-loaders-worker/
6 lines
5 lines
swh-scheduler-worker/
5 lines
6 lines

Diff 3854

View Options

dockerfiles/swh-indexer-worker/Dockerfile

Loading...
View Options

dockerfiles/swh-indexer-worker/entrypoint.sh

Loading...
View Options

dockerfiles/swh-listers-worker/Dockerfile

Loading...
View Options

dockerfiles/swh-listers-worker/entrypoint.sh

Loading...
View Options

dockerfiles/swh-loaders-worker/Dockerfile

Loading...
View Options

dockerfiles/swh-loaders-worker/entrypoint.sh

Loading...
View Options

dockerfiles/swh-scheduler-worker/Dockerfile

Loading...
View Options

dockerfiles/swh-scheduler-worker/entrypoint.sh

Loading...
About · Developer information · Legal