Page MenuHomeSoftware Heritage

Separate loader-deposit from other loaders
ClosedPublic

Authored by ardumont on Feb 17 2021, 6:40 PM.

Details

Summary

Fix the build [1]

Loaders have been reworked to only deal with configuration as constructor parameters. As
the current docker configuration is shared amongst all loaders, this can no longer work.

The current "next-gen" loaders share a subset of those configuration though so most can
run together. Except for the loader deposit which needs dedicated extra keys (deposit,
url).

Note that some configuration keys (scheduler for example) referenced in the current
configuration are not for loaders. This make instantiation fails. So they need to be
removed.

All in all, trying to separate the dedicated deposit loader with its configuration in
its own container and let the other loaders running as before fixes the build.

That and stop referecing the scheduler configuration in the loader configuration.
Instead use a dedicated environment variable to specify the scheduler url to use.

Related to T1410

[1] https://jenkins.softwareheritage.org/view/all/job/swh-docker-dev/730/console

Test Plan

tox happy (finally)

...
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  flake8: commands succeeded
  py3: commands succeeded
  shell_tests: commands succeeded
  congratulations :)

Diff Detail

Repository
rDENV Development environment
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19321
Build 29956: arc lint + arc unit

Event Timeline

Drop deposit dependency on main loaders

anlambert added a subscriber: anlambert.

Indeed, I do not see how to fix that differently.

docker/docker-compose.yml
352

Maybe it could be set to loader-deposit ?

This revision is now accepted and ready to land.Feb 17 2021, 6:59 PM
docker/docker-compose.yml
352

yes, why not.
i don't remember what that entails exactly.

Note that this is not enough, i have also error about the scheduler refusing to be instantiated [1]
Currently digging.

swh-loader_1                    | wait-for-it: swh-scheduler:5008 is available after 0 seconds
swh-loader_1                    | Traceback (most recent call last):
swh-loader_1                    |   File "/srv/softwareheritage/venv/bin/swh", line 8, in <module>
swh-loader_1                    |     sys.exit(main())
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/cli/__init__.py", line 185, in main
swh-loader_1                    |     return swh(auto_envvar_prefix="SWH")
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
swh-loader_1                    |     return self.main(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
swh-loader_1                    |     rv = self.invoke(ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1256, in invoke
swh-loader_1                    |     Command.invoke(self, ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
swh-loader_1                    |     return ctx.invoke(self.callback, **ctx.params)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
swh-loader_1                    |     return callback(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
swh-loader_1                    |     return f(get_current_context(), *args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/cli/task_type.py", line 47, in task_type
swh-loader_1                    |     raise ValueError("Scheduler class (local/remote) must be instantiated")
swh-loader_1                    | ValueError: Scheduler class (local/remote) must be instantiated
ardumont edited the summary of this revision. (Show Details)

Explicit the scheduler to use

Use an env variable to set the scheduler url instance

Fix loader-deposit.yml configuration filename typo

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Rework commit message (sync with diff)