Page MenuHomeSoftware Heritage

Separate loader-deposit from other loaders

Authored by ardumont on Wed, Feb 17, 6:40 PM.



Fix the build [1]

Loaders have been reworked to only deal with configuration as constructor parameters. As
the current docker configuration is shared amongst all loaders, this can no longer work.

The current "next-gen" loaders share a subset of those configuration though so most can
run together. Except for the loader deposit which needs dedicated extra keys (deposit,

Note that some configuration keys (scheduler for example) referenced in the current
configuration are not for loaders. This make instantiation fails. So they need to be

All in all, trying to separate the dedicated deposit loader with its configuration in
its own container and let the other loaders running as before fixes the build.

That and stop referecing the scheduler configuration in the loader configuration.
Instead use a dedicated environment variable to specify the scheduler url to use.

Related to T1410


Test Plan

tox happy (finally)

___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  flake8: commands succeeded
  py3: commands succeeded
  shell_tests: commands succeeded
  congratulations :)

Diff Detail

rDENV Development environment
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Drop deposit dependency on main loaders

anlambert added a subscriber: anlambert.

Indeed, I do not see how to fix that differently.


Maybe it could be set to loader-deposit ?

This revision is now accepted and ready to land.Wed, Feb 17, 6:59 PM

yes, why not.
i don't remember what that entails exactly.

Note that this is not enough, i have also error about the scheduler refusing to be instantiated [1]
Currently digging.

swh-loader_1                    | wait-for-it: swh-scheduler:5008 is available after 0 seconds
swh-loader_1                    | Traceback (most recent call last):
swh-loader_1                    |   File "/srv/softwareheritage/venv/bin/swh", line 8, in <module>
swh-loader_1                    |     sys.exit(main())
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/cli/", line 185, in main
swh-loader_1                    |     return swh(auto_envvar_prefix="SWH")
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 829, in __call__
swh-loader_1                    |     return self.main(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 782, in main
swh-loader_1                    |     rv = self.invoke(ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 1259, in invoke
swh-loader_1                    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 1256, in invoke
swh-loader_1                    |     Command.invoke(self, ctx)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 1066, in invoke
swh-loader_1                    |     return ctx.invoke(self.callback, **ctx.params)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 610, in invoke
swh-loader_1                    |     return callback(*args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/click/", line 21, in new_func
swh-loader_1                    |     return f(get_current_context(), *args, **kwargs)
swh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/cli/", line 47, in task_type
swh-loader_1                    |     raise ValueError("Scheduler class (local/remote) must be instantiated")
swh-loader_1                    | ValueError: Scheduler class (local/remote) must be instantiated
ardumont edited the summary of this revision. (Show Details)

Explicit the scheduler to use

Use an env variable to set the scheduler url instance

Fix loader-deposit.yml configuration filename typo

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)

Rework commit message (sync with diff)