Page MenuHomeSoftware Heritage
Differential D7632

BaseLoader: Add hook to call metadata fetchers before loading an origin
ClosedPublic
Actions

Authored by vlorentz on Apr 22 2022, 2:59 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4188: Make swh-loader-core run metadata fetchers before loading an origin
Commits
rDLDBASE584a1e4a92df: BaseLoader: Add hook to call metadata fetchers before loading an origin
Summary

This uses setuptools entrypoints to allow metadata fetchers to
make themselves discoverable without explicitly depending on them.

This commit also lifts the generic metadata loading from the base
PackageLoader to BaseLoader

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
origin-metadata
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28694
Build 44837: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 44836: arc lint + arc unit

Event Timeline

vlorentz created this revision.Apr 22 2022, 2:59 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 22 2022, 2:59 PM
swh-public-ci added a comment.Apr 22 2022, 3:00 PM

Build has FAILED

Patch application report for D7632 (id=27637)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit ae39e2e70a3fec452efe685aa736366517288747
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/727/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/727/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2022, 3:00 PM
Harbormaster failed remote builds in B28689: Diff 27637!
vlorentz updated this revision to Diff 27638.Apr 22 2022, 3:01 PM

fix

swh-public-ci added a comment.Apr 22 2022, 3:02 PM

Build has FAILED

Patch application report for D7632 (id=27638)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 20ce4c8c3e7667100e30301bbd2636974cfe23d6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/728/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/728/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 22 2022, 3:02 PM
Harbormaster failed remote builds in B28690: Diff 27638!
vlorentz updated this revision to Diff 27640.Apr 22 2022, 3:07 PM
vlorentz added a child revision: D7633: Initialize base metadata fetcher + github.

wtf mypy

swh-public-ci added a comment.Apr 22 2022, 3:10 PM

Build is green

Patch application report for D7632 (id=27640)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 49bdee98e8611a51ba1090ae0ca3f926e8e36e29
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/729/ for more details.

Harbormaster completed remote builds in B28694: Diff 27640.Apr 22 2022, 3:10 PM
vlorentz requested review of this revision.Apr 22 2022, 3:10 PM
vlorentz added a task: T4188: Make swh-loader-core run metadata fetchers before loading an origin.Apr 22 2022, 3:52 PM
vlorentz added inline comments.Apr 22 2022, 4:00 PM
swh/loader/core/loader.py
81

Should I rename lister_instance to lister_instance_name for consistency with https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/model.py$107-108 ?

vlorentz added inline comments.Apr 22 2022, 4:30 PM
swh/loader/core/loader.py
81

ah actually I *need* to do that

vlorentz updated this revision to Diff 27648.Apr 22 2022, 4:40 PM

Fix inconsistent name

swh-public-ci added a comment.Apr 22 2022, 4:43 PM

Build is green

Patch application report for D7632 (id=27648)

Rebasing onto c4c0a9b5aa...

First, rewinding head to replay your work on top of it...
Applying: BaseLoader: Add hook to call metadata fetchers before loading an origin
Changes applied before test
commit 29876a0cbdb6095a5bb10c10917db8910adfc082
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/731/ for more details.

Harbormaster completed remote builds in B28701: Diff 27648.Apr 22 2022, 4:43 PM
ardumont added a subscriber: ardumont.Apr 25 2022, 3:49 PM
ardumont added inline comments.
swh/loader/core/loader.py
85–92

or something ^?

vlorentz added inline comments.Apr 25 2022, 3:52 PM
swh/loader/core/loader.py
85–92

I did this initially, but I prefer two distinct error messages; they will make issues easier to pinpoint.

(and btw, XOR can be written more shortly in Python: if (lister_name is None) != (lister_instance_name is None):)

ardumont added inline comments.Apr 25 2022, 4:47 PM
swh/loader/core/loader.py
85–92

ack

neat xor trick ;)

ardumont accepted this revision.Apr 25 2022, 4:48 PM
This revision is now accepted and ready to land.Apr 25 2022, 4:48 PM
vlorentz updated this revision to Diff 27706.Apr 26 2022, 11:30 AM

rebase

swh-public-ci added a comment.Apr 26 2022, 11:34 AM

Build is green

Patch application report for D7632 (id=27706)

Rebasing onto c4c0a9b5aa...

Current branch diff-target is up to date.
Changes applied before test
commit 584a1e4a92df9d2d44de0da1e23b3c1ec23108d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 22 14:59:07 2022 +0200

    BaseLoader: Add hook to call metadata fetchers before loading an origin
    
    This uses setuptools entrypoints to allow metadata fetchers to
    make themselves discoverable without explicitly depending on them.
    
    This commit also lifts the generic metadata loading from the base
    PackageLoader to BaseLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/742/ for more details.

Harbormaster completed remote builds in B28763: Diff 27706.Apr 26 2022, 11:34 AM
Closed by commit rDLDBASE584a1e4a92df: BaseLoader: Add hook to call metadata fetchers before loading an origin (authored by vlorentz). · Explain WhyApr 26 2022, 11:36 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDBASE584a1e4a92df: BaseLoader: Add hook to call metadata fetchers before loading an origin.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
core/
92 lines83%
Loading...
47 lines71%
Loading...
tests/
52 lines97%
Loading...
package/
36 lines97%
Loading...

Diff 27640

View Options

swh/loader/core/loader.py

Loading...
View Options

swh/loader/core/metadata_fetchers.py

Loading...
View Options

swh/loader/core/tests/test_loader.py

Loading...
View Options

swh/loader/package/loader.py

Loading...
About · Developer information · Legal