Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T1410: Kill implicit configuration: new configuration scheme

Commits

rDENV467abd4d11b4: Separate loader-deposit from other loaders

Summary

Fix the build [1]

Loaders have been reworked to only deal with configuration as constructor parameters. As
the current docker configuration is shared amongst all loaders, this can no longer work.

The current "next-gen" loaders share a subset of those configuration though so most can
run together. Except for the loader deposit which needs dedicated extra keys (deposit,
url).

Note that some configuration keys (scheduler for example) referenced in the current
configuration are not for loaders. This make instantiation fails. So they need to be
removed.

All in all, trying to separate the dedicated deposit loader with its configuration in
its own container and let the other loaders running as before fixes the build.

That and stop referecing the scheduler configuration in the loader configuration.
Instead use a dedicated environment variable to specify the scheduler url to use.

Related to T1410

[1] https://jenkins.softwareheritage.org/view/all/job/swh-docker-dev/730/console

Test Plan

tox happy (finally)

...
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  flake8: commands succeeded
  py3: commands succeeded
  shell_tests: commands succeeded
  congratulations :)

Diff Detail

Repository

rDENV Development environment

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 19321
Build 29956: arc lint + arc unit

Event Timeline

ardumont created this revision.Feb 17 2021, 6:40 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 17 2021, 6:40 PM

ardumont requested review of this revision.Feb 17 2021, 6:40 PM

Harbormaster completed remote builds in B19314: Diff 18216.Feb 17 2021, 6:40 PM

Drop deposit dependency on main loaders

Harbormaster completed remote builds in B19315: Diff 18217.Feb 17 2021, 6:42 PM

Indeed, I do not see how to fix that differently.

docker/docker-compose.yml
352	Maybe it could be set to `loader-deposit` ?

This revision is now accepted and ready to land.Feb 17 2021, 6:59 PM

ardumont added inline comments.Feb 18 2021, 9:08 AM

docker/docker-compose.yml
352	yes, why not. i don't remember what that entails exactly.

ardumont edited the summary of this revision. (Show Details)Feb 18 2021, 9:11 AM

ardumont edited the summary of this revision. (Show Details)Feb 18 2021, 9:16 AM

Note that this is not enough, i have also error about the scheduler refusing to be instantiated [1]
Currently digging.

swh-loader_1                    | wait-for-it: swh-scheduler:5008 is available after 0 seconds
swh-loader_1                    | Traceback (most recent call last):
swh-loader_1                    |   File "/srv/softwareheritage/venv/bin/swh", line 8, in <module>
swh-loader_1                    |     sys.exit(main())
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/cli/__init__.py", line 185, in main
swh-loader_1                    |     return swh(auto_envvar_prefix="SWH")
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
swh-loader_1                    |     return self.main(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
swh-loader_1                    |     rv = self.invoke(ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1256, in invoke
swh-loader_1                    |     Command.invoke(self, ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
swh-loader_1                    |     return ctx.invoke(self.callback, **ctx.params)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
swh-loader_1                    |     return callback(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
swh-loader_1                    |     return f(get_current_context(), *args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/cli/task_type.py", line 47, in task_type
swh-loader_1                    |     raise ValueError("Scheduler class (local/remote) must be instantiated")
swh-loader_1                    | ValueError: Scheduler class (local/remote) must be instantiated