Page MenuHomeSoftware Heritage

BaseLoader: Add hook to call metadata fetchers before loading an origin
ClosedPublic

Authored by vlorentz on Apr 22 2022, 2:59 PM.

Details

Summary

This uses setuptools entrypoints to allow metadata fetchers to
make themselves discoverable without explicitly depending on them.

This commit also lifts the generic metadata loading from the base
PackageLoader to BaseLoader

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build has FAILED

Patch application report for D7632 (id=27637)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit ae39e2e70a3fec452efe685aa736366517288747
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/727/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/727/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2022, 3:00 PM
Harbormaster failed remote builds in B28689: Diff 27637!

Build has FAILED

Patch application report for D7632 (id=27638)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 20ce4c8c3e7667100e30301bbd2636974cfe23d6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/728/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/728/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2022, 3:02 PM
Harbormaster failed remote builds in B28690: Diff 27638!

Build is green

Patch application report for D7632 (id=27640)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 49bdee98e8611a51ba1090ae0ca3f926e8e36e29
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/729/ for more details.

swh/loader/core/loader.py
80

Should I rename lister_instance to lister_instance_name for consistency with https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/model.py$107-108 ?

swh/loader/core/loader.py
80

ah actually I *need* to do that

Build is green

Patch application report for D7632 (id=27648)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 29876a0cbdb6095a5bb10c10917db8910adfc082
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/731/ for more details.

ardumont added inline comments.
swh/loader/core/loader.py
85–92

or something ^?

swh/loader/core/loader.py
85–92

I did this initially, but I prefer two distinct error messages; they will make issues easier to pinpoint.

(and btw, XOR can be written more shortly in Python: if (lister_name is None) != (lister_instance_name is None):)

swh/loader/core/loader.py
85–92

ack

neat xor trick ;)

This revision is now accepted and ready to land.Apr 25 2022, 4:48 PM

Build is green

Patch application report for D7632 (id=27706)

Rebasing onto c4c0a9b5aa...

Current branch diff-target is up to date.
Changes applied before test
commit 584a1e4a92df9d2d44de0da1e23b3c1ec23108d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/742/ for more details.