Page MenuHomeSoftware Heritage
Paste P1205

docker-compose no longer allows override on containers
ActivePublic
Actions

Authored by ardumont on Oct 21 2021, 4:16 PM.
Looks like override no longer works for me:
```
$ docker-compose --version
docker-compose version 1.29.2, build unknown
$ docker --version
Docker version 20.10.5+dfsg1, build 55c4c88
$ which docker
/usr/bin/docker
$ swh-doco up -d
...
$ swh-doco logs -f swh-scheduler-runner
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml logs -f swh-scheduler-runner
Attaching to docker_swh-scheduler-runner_1
swh-scheduler-runner_1 | Using pip from /srv/softwareheritage/venv/bin/pip
swh-scheduler-runner_1 | /src /
swh-scheduler-runner_1 | Obtaining file:///src/swh-scheduler
swh-scheduler-runner_1 | Installing build dependencies: started
swh-scheduler-runner_1 | Installing build dependencies: finished with status 'done'
swh-scheduler-runner_1 | Getting requirements to build editable: started
swh-scheduler-runner_1 | Getting requirements to build editable: finished with status 'done'
swh-scheduler-runner_1 | Preparing editable metadata (pyproject.toml): started
swh-scheduler-runner_1 | Preparing editable metadata (pyproject.toml): finished with status 'done'
swh-scheduler-runner_1 | Build backend does not support editables, falling back to setup.py egg_info.
swh-scheduler-runner_1 | Preparing metadata (setup.py): started
swh-scheduler-runner_1 | Preparing metadata (setup.py): finished with status 'error'
swh-scheduler-runner_1 | ERROR: Command errored out with exit status 1:
swh-scheduler-runner_1 | command: /srv/softwareheritage/venv/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/src/swh-scheduler/setup.py'"'"'; __file__='"'"'/src/swh-scheduler/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-k13v_bef
swh-scheduler-runner_1 | cwd: /src/swh-scheduler/
swh-scheduler-runner_1 | Complete output (31 lines):
swh-scheduler-runner_1 | /srv/softwareheritage/venv/bin/python3: No module named pip
swh-scheduler-runner_1 | Traceback (most recent call last):
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
swh-scheduler-runner_1 | subprocess.check_call(cmd)
swh-scheduler-runner_1 | File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
swh-scheduler-runner_1 | raise CalledProcessError(retcode, cmd)
swh-scheduler-runner_1 | subprocess.CalledProcessError: Command '['/srv/softwareheritage/venv/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpm_p5lqa0', '--quiet', 'setuptools-scm']' returned non-zero exit status 1.
swh-scheduler-runner_1 |
swh-scheduler-runner_1 | The above exception was the direct cause of the following exception:
swh-scheduler-runner_1 |
swh-scheduler-runner_1 | Traceback (most recent call last):
swh-scheduler-runner_1 | File "<string>", line 1, in <module>
swh-scheduler-runner_1 | File "/src/swh-scheduler/setup.py", line 74, in <module>
swh-scheduler-runner_1 | "Documentation": "https://docs.softwareheritage.org/devel/swh-scheduler/",
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
swh-scheduler-runner_1 | _install_setup_requires(attrs)
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
swh-scheduler-runner_1 | dist.fetch_build_eggs(dist.setup_requires)
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 809, in fetch_build_eggs
swh-scheduler-runner_1 | replace_conflicting=True,
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
swh-scheduler-runner_1 | replace_conflicting=replace_conflicting
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
swh-scheduler-runner_1 | return self.obtain(req, installer)
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
swh-scheduler-runner_1 | return installer(requirement)
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 877, in fetch_build_egg
swh-scheduler-runner_1 | return fetch_build_egg(self, req)
swh-scheduler-runner_1 | File "/tmp/pip-build-env-9hd4jvce/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
swh-scheduler-runner_1 | raise DistutilsError(str(e)) from e
swh-scheduler-runner_1 | distutils.errors.DistutilsError: Command '['/srv/softwareheritage/venv/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpm_p5lqa0', '--quiet', 'setuptools-scm']' returned non-zero exit status 1.
swh-scheduler-runner_1 | ----------------------------------------
swh-scheduler-runner_1 | WARNING: Discarding file:///src/swh-scheduler. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
swh-scheduler-runner_1 | ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
docker_swh-scheduler-runner_1 exited with code 1
```
with docker-compose.override.yml:
```
swh-scheduler-runner:
volumes:
- "$SWH_ENVIRONMENT_HOME/swh-scheduler:/src/swh-scheduler"
```
Without ^, all is fine though.

Event Timeline

ardumont created this paste.Oct 21 2021, 4:16 PM
ardumont changed the title of this paste from untitled to docker-compose no longer allows me to override some containers.Oct 21 2021, 4:43 PM
ardumont edited the content of this paste. (Show Details)
ardumont added a comment.EditedOct 22 2021, 10:20 AM

After patching the swh-environment/docker/dockerfile:

# RUN pip install --upgrade pip setuptools wheel
RUN pip install setuptools wheel

maybe related to what's been done in 2af28729729d7aa16fb63ecf79c9303f26c09d17

This actually goes beyond the actual problem, and fails on another problem:

=========================================================================================================== short test summary info ===========================================================================================================
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-pa...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/runner.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_celery_tasks.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_celery_tasks.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages), R...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages), R...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================================== 1 warning, 9 errors in 4.16s =========================================================================================================
ERROR: InvocationError for command /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/bin/pytest --doctest-modules --hypothesis-profile=fast --cov=/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages/swh/scheduler .tox/py3/lib/python3.9/site-packages/swh/scheduler --cov-branch (exited with code 2)
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  black: commands succeeded
  flake8: commands succeeded
ERROR:   mypy: commands failed
ERROR:   py3: commands failed

which is somehow consistent with another error i got when using tox within the scheduler

=========================================================================================================== short test summary info ===========================================================================================================
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-pa...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/celery_backend/runner.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_celery_tasks.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_celery_tasks.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages), R...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_config.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages), R...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
ERROR .tox/py3/lib/python3.9/site-packages/swh/scheduler/tests/test_orchestrator.py - pkg_resources.ContextualVersionConflict: (click 8.0.3 (/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packag...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================================== 1 warning, 9 errors in 4.16s =========================================================================================================
ERROR: InvocationError for command /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/bin/pytest --doctest-modules --hypothesis-profile=fast --cov=/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/py3/lib/python3.9/site-packages/swh/scheduler .tox/py3/lib/python3.9/site-packages/swh/scheduler --cov-branch (exited with code 2)
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  black: commands succeeded
  flake8: commands succeeded
ERROR:   mypy: commands failed
ERROR:   py3: commands failed
ardumont mentioned this in D6536: Restrict the click version to avoid conflict version with celery's.Oct 22 2021, 10:52 AM
ardumont mentioned this in D6537: Dockerfile: Avoid pip 21.3 version preventing working container override.Oct 22 2021, 11:32 AM
ardumont mentioned this in T3685: Our pip usage is broken since a recent pip release.Oct 22 2021, 11:33 AM
ardumont mentioned this in rDSCHecc0e2803e0b: Restrict the click version to avoid conflict version with celery's.Oct 22 2021, 11:34 AM
ardumont changed the title of this paste from docker-compose no longer allows me to override some containers to docker-compose no longer allows override on containers.Oct 22 2021, 11:35 AM
About · Developer information · Legal