Page MenuHomeSoftware Heritage

core.loader: Merge Loader into BaseLoader
ClosedPublic

Authored by ardumont on Feb 16 2021, 6:49 PM.

Details

Summary

This simplifies the docstring to a more relevant docstring.

This drops the spurious {*args, **kwargs} no longer used in the core loaders.
(it's required because PackageLoader's load signature is without (kw)args)

Depends on D5071

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
update-config
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19272
Build 29889: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 29888: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D5088 (id=18163)

Could not rebase; Attempt merge onto 7ea69e2b70...

Updating 7ea69e2..593641a
Fast-forward
 swh/loader/cli.py                                |  33 ++++-
 swh/loader/core/loader.py                        | 133 ++++++++++---------
 swh/loader/core/tests/test_loader.py             |  47 ++++---
 swh/loader/package/archive/loader.py             |   7 +-
 swh/loader/package/archive/tasks.py              |   5 +-
 swh/loader/package/archive/tests/test_archive.py |  74 ++++++-----
 swh/loader/package/archive/tests/test_tasks.py   |  12 +-
 swh/loader/package/cran/loader.py                |  13 +-
 swh/loader/package/cran/tasks.py                 |   4 +-
 swh/loader/package/cran/tests/test_cran.py       |  42 +++---
 swh/loader/package/cran/tests/test_tasks.py      |  17 ++-
 swh/loader/package/debian/loader.py              |  14 +-
 swh/loader/package/debian/tasks.py               |   3 +-
 swh/loader/package/debian/tests/test_debian.py   |  65 +++++----
 swh/loader/package/debian/tests/test_tasks.py    |  12 +-
 swh/loader/package/deposit/loader.py             |  35 ++++-
 swh/loader/package/deposit/tasks.py              |   4 +-
 swh/loader/package/deposit/tests/conftest.py     |   9 +-
 swh/loader/package/deposit/tests/test_deposit.py |  45 +++++--
 swh/loader/package/deposit/tests/test_tasks.py   |  15 ++-
 swh/loader/package/loader.py                     |  39 ++----
 swh/loader/package/nixguix/loader.py             |  16 ++-
 swh/loader/package/nixguix/tasks.py              |   4 +-
 swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++--------
 swh/loader/package/nixguix/tests/test_tasks.py   |  20 +--
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/npm/tasks.py                  |   4 +-
 swh/loader/package/npm/tests/test_npm.py         |  89 ++++++-------
 swh/loader/package/npm/tests/test_tasks.py       |  10 +-
 swh/loader/package/pypi/loader.py                |  10 +-
 swh/loader/package/pypi/tasks.py                 |   4 +-
 swh/loader/package/pypi/tests/test_pypi.py       | 160 ++++++++++-------------
 swh/loader/package/pypi/tests/test_tasks.py      |  10 +-
 swh/loader/package/tests/test_loader.py          |   8 +-
 swh/loader/package/tests/test_loader_metadata.py |  47 ++-----
 swh/loader/pytest_plugin.py                      |  28 ++--
 swh/loader/tests/conftest.py                     |   4 +-
 swh/loader/tests/test_cli.py                     |  65 +++++----
 swh/loader/tests/test_init.py                    |  11 +-
 39 files changed, 674 insertions(+), 568 deletions(-)
Changes applied before test
commit 593641af0eaab910cb9a82c7180b8d08dbb1eb45
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Feb 16 18:48:42 2021 +0100

    core.loader: Merge Loader into BaseLoader
    
    This also simplifies the docstring to a more relevant docstring.
    
    This also drops the spurious {*args, **kwargs} no longer used in the core loaders.

commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 09:04:17 2021 +0100

    Unify loader instantiation
    
    This unifies and centralizes the instantiation the same way the lister does.
    
    This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only
    concern for now is to instantiate loaders from either a configuration dict or a
    configuration file.
    
    This simplify instantiation in celery task code and avoids duplicating the configuration
    load in each loader constructor.
    
    The end goal is to simplify the future refactoring on configuration. Now, we will only
    have to adapt the Loader class when we finally start simplifying uniformly the
    configuration.

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/376/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 16 2021, 6:50 PM
Harbormaster failed remote builds in B19267: Diff 18163!

Fix format

(my black is dead ¯\_(ツ)_/¯)

Build is green

Patch application report for D5088 (id=18164)

Could not rebase; Attempt merge onto 7ea69e2b70...

Updating 7ea69e2..3bb2cfd
Fast-forward
 swh/loader/cli.py                                |  33 ++++-
 swh/loader/core/loader.py                        | 138 ++++++++++---------
 swh/loader/core/tests/test_loader.py             |  47 ++++---
 swh/loader/package/archive/loader.py             |   7 +-
 swh/loader/package/archive/tasks.py              |   5 +-
 swh/loader/package/archive/tests/test_archive.py |  74 ++++++-----
 swh/loader/package/archive/tests/test_tasks.py   |  12 +-
 swh/loader/package/cran/loader.py                |  13 +-
 swh/loader/package/cran/tasks.py                 |   4 +-
 swh/loader/package/cran/tests/test_cran.py       |  42 +++---
 swh/loader/package/cran/tests/test_tasks.py      |  17 ++-
 swh/loader/package/debian/loader.py              |  14 +-
 swh/loader/package/debian/tasks.py               |   3 +-
 swh/loader/package/debian/tests/test_debian.py   |  65 +++++----
 swh/loader/package/debian/tests/test_tasks.py    |  12 +-
 swh/loader/package/deposit/loader.py             |  35 ++++-
 swh/loader/package/deposit/tasks.py              |   4 +-
 swh/loader/package/deposit/tests/conftest.py     |   9 +-
 swh/loader/package/deposit/tests/test_deposit.py |  45 +++++--
 swh/loader/package/deposit/tests/test_tasks.py   |  15 ++-
 swh/loader/package/loader.py                     |  39 ++----
 swh/loader/package/nixguix/loader.py             |  16 ++-
 swh/loader/package/nixguix/tasks.py              |   4 +-
 swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++--------
 swh/loader/package/nixguix/tests/test_tasks.py   |  20 +--
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/npm/tasks.py                  |   4 +-
 swh/loader/package/npm/tests/test_npm.py         |  89 ++++++-------
 swh/loader/package/npm/tests/test_tasks.py       |  10 +-
 swh/loader/package/pypi/loader.py                |  10 +-
 swh/loader/package/pypi/tasks.py                 |   4 +-
 swh/loader/package/pypi/tests/test_pypi.py       | 160 ++++++++++-------------
 swh/loader/package/pypi/tests/test_tasks.py      |  10 +-
 swh/loader/package/tests/test_loader.py          |   8 +-
 swh/loader/package/tests/test_loader_metadata.py |  47 ++-----
 swh/loader/pytest_plugin.py                      |  28 ++--
 swh/loader/tests/conftest.py                     |   4 +-
 swh/loader/tests/test_cli.py                     |  65 +++++----
 swh/loader/tests/test_init.py                    |  11 +-
 39 files changed, 679 insertions(+), 568 deletions(-)
Changes applied before test
commit 3bb2cfda4ee5ad78a928159f0dc994d9307395fe
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Feb 16 18:48:42 2021 +0100

    core.loader: Merge Loader into BaseLoader
    
    This also simplifies the docstring to a more relevant docstring.
    
    This also drops the spurious {*args, **kwargs} no longer used in the core loaders.

commit 0c36690f10b26a9916150a5606d3de2ad2d97cb6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 09:04:17 2021 +0100

    Unify loader instantiation
    
    This unifies and centralizes the instantiation the same way the lister does.
    
    This introduces a new base class `swh.loader.pattern.Loader` for all loaders whose only
    concern for now is to instantiate loaders from either a configuration dict or a
    configuration file.
    
    This simplify instantiation in celery task code and avoids duplicating the configuration
    load in each loader constructor.
    
    The end goal is to simplify the future refactoring on configuration. Now, we will only
    have to adapt the Loader class when we finally start simplifying uniformly the
    configuration.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/377/ for more details.

  • Rework class docstring
  • Rebase

Build is green

Patch application report for D5088 (id=18168)

Could not rebase; Attempt merge onto 7ea69e2b70...

Updating 7ea69e2..7c776a4
Fast-forward
 swh/loader/cli.py                                |  33 ++++-
 swh/loader/core/loader.py                        | 142 +++++++++++---------
 swh/loader/core/tests/test_loader.py             |  47 ++++---
 swh/loader/package/archive/loader.py             |   7 +-
 swh/loader/package/archive/tasks.py              |   5 +-
 swh/loader/package/archive/tests/test_archive.py |  74 ++++++-----
 swh/loader/package/archive/tests/test_tasks.py   |  12 +-
 swh/loader/package/cran/loader.py                |  13 +-
 swh/loader/package/cran/tasks.py                 |   4 +-
 swh/loader/package/cran/tests/test_cran.py       |  42 +++---
 swh/loader/package/cran/tests/test_tasks.py      |  17 ++-
 swh/loader/package/debian/loader.py              |  14 +-
 swh/loader/package/debian/tasks.py               |   3 +-
 swh/loader/package/debian/tests/test_debian.py   |  65 +++++----
 swh/loader/package/debian/tests/test_tasks.py    |  12 +-
 swh/loader/package/deposit/loader.py             |  35 ++++-
 swh/loader/package/deposit/tasks.py              |   4 +-
 swh/loader/package/deposit/tests/conftest.py     |   9 +-
 swh/loader/package/deposit/tests/test_deposit.py |  45 +++++--
 swh/loader/package/deposit/tests/test_tasks.py   |  15 ++-
 swh/loader/package/loader.py                     |  39 ++----
 swh/loader/package/nixguix/loader.py             |  16 ++-
 swh/loader/package/nixguix/tasks.py              |   4 +-
 swh/loader/package/nixguix/tests/test_nixguix.py | 114 ++++++++--------
 swh/loader/package/nixguix/tests/test_tasks.py   |  20 +--
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/npm/tasks.py                  |   4 +-
 swh/loader/package/npm/tests/test_npm.py         |  89 ++++++-------
 swh/loader/package/npm/tests/test_tasks.py       |  10 +-
 swh/loader/package/pypi/loader.py                |  10 +-
 swh/loader/package/pypi/tasks.py                 |   4 +-
 swh/loader/package/pypi/tests/test_pypi.py       | 160 ++++++++++-------------
 swh/loader/package/pypi/tests/test_tasks.py      |  10 +-
 swh/loader/package/tests/test_loader.py          |   8 +-
 swh/loader/package/tests/test_loader_metadata.py |  47 ++-----
 swh/loader/pytest_plugin.py                      |  28 ++--
 swh/loader/tests/conftest.py                     |   4 +-
 swh/loader/tests/test_cli.py                     |  65 +++++----
 swh/loader/tests/test_init.py                    |  11 +-
 39 files changed, 682 insertions(+), 569 deletions(-)
Changes applied before test
commit 7c776a487c648d8e0a804783408f6d954b4f84ef
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Feb 16 18:48:42 2021 +0100

    core.loader: Merge Loader into BaseLoader
    
    This also simplifies the docstring to a more relevant docstring.
    
    This also drops the spurious {*args, **kwargs} no longer used in the core loaders.

commit 7116bb75897ae878f68679e86aaa5aec6cc507be
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Feb 8 09:04:17 2021 +0100

    Unify loader instantiation
    
    This unifies and centralizes the instantiation the same way the lister does.
    
    This introduces a new base class swh.loader.core.loader.Loader for all loaders whose
    only concern for now is to instantiate loaders from either a configuration dict or a
    configuration file.
    
    This simplifies instantiation in celery task code and avoids duplicating the
    configuration load in each loader constructor.
    
    The end goal is to simplify the future refactoring on configuration. With the following,
    we will only have to adapt the Loader class when we start simplifying uniformly the
    configuration.
    
    Also note that I mostly reused the equivalent `swh.lister.pattern.Lister.from_config*`.
    I did not refactor the common behavior (to avoid throwing another dependency in the
    mix). That could always be refactored later.
    
    (inspired by both the work on listers and the configuration system work)
    
    Related to T1410

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/379/ for more details.

olasd added a subscriber: olasd.

We need to make sure that other loaders really don't use the *args, **kwargs, but yeah, that looks fine.

Did you run mypy on other swh-loader-* modules against this change?

This revision is now accepted and ready to land.Feb 17 2021, 9:40 AM

We need to make sure that other loaders really don't use the *args, **kwargs, but yeah, that looks fine.

I checked git and svn so far.
I'll check mercurial prior to land it.

Did you run mypy on other swh-loader-* modules against this change?

nope, not yet. Just mostly read the code.

Did you run mypy on other swh-loader-* modules against this change?

nope, not yet. Just mostly read the code.

I adapted locally the vcs loaders depending on this and everything went fine after this.
I'll land this and amend the 3 diffs on loaders with this now.