Page MenuHomeSoftware Heritage
Create Task
Maniphest T2080

API servers crash on startup in docker
Closed, MigratedEdits Locked
Actions

Description

swh-scheduler-api_1                  | Traceback (most recent call last):
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/util.py", line 340, in import_app
swh-scheduler-api_1                  |     app = getattr(mod, obj)
swh-scheduler-api_1                  | AttributeError: module 'swh.scheduler.api.server' has no attribute 'make_app_from_configfile()'
swh-scheduler-api_1                  | 
swh-scheduler-api_1                  | During handling of the above exception, another exception occurred:
swh-scheduler-api_1                  | 
swh-scheduler-api_1                  | Traceback (most recent call last):
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
swh-scheduler-api_1                  |     worker.init_process()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
swh-scheduler-api_1                  |     super().init_process()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 133, in init_process
swh-scheduler-api_1                  |     self.load_wsgi()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
swh-scheduler-api_1                  |     self.wsgi = self.app.wsgi()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
swh-scheduler-api_1                  |     self.callable = self.load()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
swh-scheduler-api_1                  |     return self.load_wsgiapp()
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
swh-scheduler-api_1                  |     return util.import_app(self.app_uri)
swh-scheduler-api_1                  |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/util.py", line 344, in import_app
swh-scheduler-api_1                  |     raise AppImportError("Failed to find application object %r in %r" % (obj, module))
swh-scheduler-api_1                  | gunicorn.errors.AppImportError: Failed to find application object 'make_app_from_configfile()' in 'swh.scheduler.api.server'
swh-scheduler-api_1                  | Failed to find application object 'make_app_from_configfile()' in 'swh.scheduler.api.server'
swh-web_1                            | Failed to find application object 'get_wsgi_application()' in 'django.core.wsgi'
swh-web_1                            | [2019-11-12 13:03:30 +0000] [19] [INFO] Worker exiting (pid: 19)
swh-web_1                            | Failed to find application object 'get_wsgi_application()' in 'django.core.wsgi'
swh-web_1                            | [2019-11-12 13:03:30 +0000] [20] [INFO] Worker exiting (pid: 20)
swh-web_1                            | Traceback (most recent call last):
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 209, in run
swh-web_1                            |     self.sleep()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 357, in sleep
swh-web_1                            |     ready = select.select([self.PIPE[0]], [], [], 1.0)
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
swh-web_1                            |     self.reap_workers()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 528, in reap_workers
swh-web_1                            |     raise HaltServer(reason, self.APP_LOAD_ERROR)
swh-web_1                            | gunicorn.errors.HaltServer: <HaltServer 'App failed to load.' 4>
swh-web_1                            | 
swh-web_1                            | During handling of the above exception, another exception occurred:
swh-web_1                            | 
swh-web_1                            | Traceback (most recent call last):
swh-web_1                            |   File "/srv/softwareheritage/venv/bin/gunicorn", line 8, in <module>
swh-web_1                            |     sys.exit(run())
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in run
swh-web_1                            |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 219, in run
swh-web_1                            |     super().run()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/app/base.py", line 72, in run
swh-web_1                            |     Arbiter(self).run()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 229, in run
swh-web_1                            |     self.halt(reason=inst.reason, exit_status=inst.exit_status)
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 342, in halt
swh-web_1                            |     self.stop()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 393, in stop
swh-web_1                            |     time.sleep(0.1)
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
swh-web_1                            |     self.reap_workers()
swh-web_1                            |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/arbiter.py", line 528, in reap_workers
swh-web_1                            |     raise HaltServer(reason, self.APP_LOAD_ERROR)
swh-web_1                            | gunicorn.errors.HaltServer: <HaltServer 'App failed to load.' 4>
swh-deposit_1                        | Traceback (most recent call last):
swh-deposit_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/gunicorn/util.py", line 340, in import_app
swh-deposit_1                        |     app = getattr(mod, obj)
swh-deposit_1                        | AttributeError: module 'django.core.wsgi' has no attribute 'get_wsgi_application()'

etc.

Affected: swh-scheduler-api, swh-storage, swh-objstorage, swh-web, swh-deposit

Revisions and Commits

rCDFD Dockerfiles for developers
D2252rCDFD8ecaa9b62838 Dockerfile: Pin gunicorn version
rDENV Development environment
D2252rDENV8ecaa9b62838 Dockerfile: Pin gunicorn version

Related Objects

Event Timeline

vlorentz triaged this task as High priority.Nov 12 2019, 2:18 PM
vlorentz created this task.
vlorentz updated the task description. (Show Details)Nov 12 2019, 2:23 PM
vlorentz updated the task description. (Show Details)
anlambert added a subscriber: anlambert.Nov 12 2019, 2:25 PM

I was also looking at it.

This is due to the recent release of gunicorn 20.0.0, more details here: https://github.com/benoitc/gunicorn/issues/2159

The swh servers fail to start because of that change in gunicorn source code.

olasd added a subscriber: olasd.Nov 12 2019, 2:28 PM

Works fine for me? Are you sure your docker image is up to date?

olasd added a comment.Nov 12 2019, 2:28 PM

Ah, I guess that's the opposite then.

anlambert added a comment.Nov 12 2019, 2:30 PM

Seems the simplest solution is to instantiate the wsgi application in swh.*.api.server modules.

For instance in swh.objstorage.api.server, adding:

app = make_app_from_config_file()

The objstorage server can then be launched with:

gunicorn --bind 0.0.0.0:5003 \
         --worker-class aiohttp.worker.GunicornWebWorker \
         --log-level DEBUG \
         --threads 4 \
         --workers 2 \
         --reload \
         --timeout 3600 \
         'swh.objstorage.api.server:app'
olasd added a comment.Nov 12 2019, 2:32 PM

We've migrated away from doing that ages ago in favor of app factories, and I'd very much prefer we avoid introducing this again: Side effects such as reading configuration files unconditionally running in importable modules are very bad form.

I suggest we make the dependency on 'gunicorn<20' explicit in the Dockerfile until they fix this mess upstream.

anlambert added a comment.Nov 12 2019, 2:36 PM

Indeed, the Dockerfile fix is much simpler. Let's hope they will fix the issue upstream (not sure regarding this comment).

anlambert mentioned this in D2252: Dockerfile: Pin gunicorn version.Nov 12 2019, 2:50 PM
anlambert closed this task as Resolved by committing rCDFD8ecaa9b62838: Dockerfile: Pin gunicorn version.Nov 12 2019, 2:57 PM
anlambert claimed this task.
anlambert added a commit: rCDFD8ecaa9b62838: Dockerfile: Pin gunicorn version.
ardumont mentioned this in D2268: Add swh-search to docker env.Nov 13 2019, 5:16 PM
anlambert added a commit: rDENV8ecaa9b62838: Dockerfile: Pin gunicorn version.Nov 26 2019, 1:35 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:28 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal