This simplifies the docstring to a more relevant docstring.
This drops the spurious {*args, **kwargs} no longer used in the core loaders.
(it's required because PackageLoader's load signature is without (kw)args)
Depends on D5071
Details
Details
- Reviewers
olasd - Group Reviewers
Reviewers - Commits
- rDLDBASE7c776a487c64: core.loader: Merge Loader into BaseLoader
Diff Detail
Diff Detail
- Repository
- rDLDBASE Generic VCS/Package Loader
- Branch
- update-config
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 19267 Build 29881: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 29880: arc lint + arc unit
Event Timeline
Comment Actions
Build has FAILED
Patch application report for D5088 (id=18163)
Could not rebase; Attempt merge onto 7ea69e2b70...
Updating 7ea69e2..593641a Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 133 ++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 674 insertions(+), 568 deletions(-)
Changes applied before test
commit 593641af0eaab910cb9a82c7180b8d08dbb1eb45
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Tue Feb 16 18:48:42 2021 +0100
core.loader: Merge Loader into BaseLoader
This also simplifies the docstring to a more relevant docstring.
This also drops the spurious {*args, **kwargs} no longer used in the core loaders.
commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon Feb 8 09:04:17 2021 +0100
Unify loader instantiation
This unifies and centralizes the instantiation the same way the lister does.
This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only
concern for now is to instantiate loaders from either a configuration dict or a
configuration file.
This simplify instantiation in celery task code and avoids duplicating the configuration
load in each loader constructor.
The end goal is to simplify the future refactoring on configuration. Now, we will only
have to adapt the Loader class when we finally start simplifying uniformly the
configuration.Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/console
Comment Actions
Build is green
Patch application report for D5088 (id=18164)
Could not rebase; Attempt merge onto 7ea69e2b70...
Updating 7ea69e2..3bb2cfd Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 138 ++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 679 insertions(+), 568 deletions(-)
Changes applied before test
commit 3bb2cfda4ee5ad78a928159f0dc994d9307395fe
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Tue Feb 16 18:48:42 2021 +0100
core.loader: Merge Loader into BaseLoader
This also simplifies the docstring to a more relevant docstring.
This also drops the spurious {*args, **kwargs} no longer used in the core loaders.
commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon Feb 8 09:04:17 2021 +0100
Unify loader instantiation
This unifies and centralizes the instantiation the same way the lister does.
This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only
concern for now is to instantiate loaders from either a configuration dict or a
configuration file.
This simplify instantiation in celery task code and avoids duplicating the configuration
load in each loader constructor.
The end goal is to simplify the future refactoring on configuration. Now, we will only
have to adapt the Loader class when we finally start simplifying uniformly the
configuration.See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/377/ for more details.
Comment Actions
Build is green
Patch application report for D5088 (id=18168)
Could not rebase; Attempt merge onto 7ea69e2b70...
Updating 7ea69e2..7c776a4 Fast-forward swh/loader/cli.py | 33 ++++- swh/loader/core/loader.py | 142 +++++++++++--------- swh/loader/core/tests/test_loader.py | 47 ++++--- swh/loader/package/archive/loader.py | 7 +- swh/loader/package/archive/tasks.py | 5 +- swh/loader/package/archive/tests/test_archive.py | 74 ++++++----- swh/loader/package/archive/tests/test_tasks.py | 12 +- swh/loader/package/cran/loader.py | 13 +- swh/loader/package/cran/tasks.py | 4 +- swh/loader/package/cran/tests/test_cran.py | 42 +++--- swh/loader/package/cran/tests/test_tasks.py | 17 ++- swh/loader/package/debian/loader.py | 14 +- swh/loader/package/debian/tasks.py | 3 +- swh/loader/package/debian/tests/test_debian.py | 65 +++++---- swh/loader/package/debian/tests/test_tasks.py | 12 +- swh/loader/package/deposit/loader.py | 35 ++++- swh/loader/package/deposit/tasks.py | 4 +- swh/loader/package/deposit/tests/conftest.py | 9 +- swh/loader/package/deposit/tests/test_deposit.py | 45 +++++-- swh/loader/package/deposit/tests/test_tasks.py | 15 ++- swh/loader/package/loader.py | 39 ++---- swh/loader/package/nixguix/loader.py | 16 ++- swh/loader/package/nixguix/tasks.py | 4 +- swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++-------- swh/loader/package/nixguix/tests/test_tasks.py | 20 +-- swh/loader/package/npm/loader.py | 10 +- swh/loader/package/npm/tasks.py | 4 +- swh/loader/package/npm/tests/test_npm.py | 89 ++++++------- swh/loader/package/npm/tests/test_tasks.py | 10 +- swh/loader/package/pypi/loader.py | 10 +- swh/loader/package/pypi/tasks.py | 4 +- swh/loader/package/pypi/tests/test_pypi.py | 160 ++++++++++------------- swh/loader/package/pypi/tests/test_tasks.py | 10 +- swh/loader/package/tests/test_loader.py | 8 +- swh/loader/package/tests/test_loader_metadata.py | 47 ++----- swh/loader/pytest_plugin.py | 28 ++-- swh/loader/tests/conftest.py | 4 +- swh/loader/tests/test_cli.py | 65 +++++---- swh/loader/tests/test_init.py | 11 +- 39 files changed, 682 insertions(+), 569 deletions(-)
Changes applied before test
commit 7c776a487c648d8e0a804783408f6d954b4f84ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Tue Feb 16 18:48:42 2021 +0100
core.loader: Merge Loader into BaseLoader
This also simplifies the docstring to a more relevant docstring.
This also drops the spurious {*args, **kwargs} no longer used in the core loaders.
commit 7116bb75897ae878f68679e86aaa5aec6cc507be
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Mon Feb 8 09:04:17 2021 +0100
Unify loader instantiation
This unifies and centralizes the instantiation the same way the lister does.
This introduces a new base class swh.loader.core.loader.Loader for all loaders whose
only concern for now is to instantiate loaders from either a configuration dict or a
configuration file.
This simplifies instantiation in celery task code and avoids duplicating the
configuration load in each loader constructor.
The end goal is to simplify the future refactoring on configuration. With the following,
we will only have to adapt the Loader class when we start simplifying uniformly the
configuration.
Also note that I mostly reused the equivalent `swh.lister.pattern.Lister.from_config*`.
I did not refactor the common behavior (to avoid throwing another dependency in the
mix). That could always be refactored later.
(inspired by both the work on listers and the configuration system work)
Related to T1410See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/379/ for more details.
Comment Actions
We need to make sure that other loaders really don't use the *args, **kwargs, but yeah, that looks fine.
Did you run mypy on other swh-loader-* modules against this change?
Comment Actions
We need to make sure that other loaders really don't use the *args, **kwargs, but yeah, that looks fine.
I checked git and svn so far.
I'll check mercurial prior to land it.
Did you run mypy on other swh-loader-* modules against this change?
nope, not yet. Just mostly read the code.
Comment Actions
Did you run mypy on other swh-loader-* modules against this change?
nope, not yet. Just mostly read the code.
I adapted locally the vcs loaders depending on this and everything went fine after this.
I'll land this and amend the 3 diffs on loaders with this now.