Page MenuHomeSoftware Heritage
Differential D7663

Add method get_parent_origins()
ClosedPublic
Actions

Authored by vlorentz on Apr 26 2022, 1:28 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T3273: Use "fork" relationships to speed-up initial load of large repositories
Commits
rDLDMD01d34fa76821: Add method get_parent_origins()
Summary

This will allow the Git loader to incrementally load GitHub forks

Diff Detail

Repository
rDLDMD Extrinsic Metadata Loaders
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Apr 26 2022, 1:28 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 26 2022, 1:28 PM
swh-public-ci added a comment.Apr 26 2022, 1:31 PM

Build is green

Patch application report for D7663 (id=27728)

Rebasing onto aa0af83a5e...

Current branch diff-target is up to date.
Changes applied before test
commit 427d6cdb5f72b9955023b81c6e060f7f2d4cd1c5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Apr 26 13:28:37 2022 +0200

    Add method get_parent_origin()
    
    This will allow the Git loader to incrementally load GitHub forks

See https://jenkins.softwareheritage.org/job/DLDMD/job/tests-on-diff/6/ for more details.

Harbormaster completed remote builds in B28787: Diff 27728.Apr 26 2022, 1:31 PM
vlorentz requested review of this revision.Apr 26 2022, 1:31 PM
olasd added a subscriber: olasd.Apr 26 2022, 4:28 PM

Should we be using the parent, or the source (which, afaik, is the root of all forks) repo? Or both?

Ideally we'd have a way to attempt all known parent repos in succession, from the closest to the farthest

swh/loader/metadata/github.py
79–81

Maybe we should build that from clone_url (stripping the .git ending if it's there) instead. I guess it's consistent with the way the lister does it?

vlorentz updated this revision to Diff 27788.Apr 26 2022, 5:32 PM
vlorentz marked an inline comment as done.
  • use html_url like listers, instead of building the URL
  • support both parent and source
swh-public-ci added a comment.Apr 26 2022, 5:35 PM

Build is green

Patch application report for D7663 (id=27788)

Rebasing onto aa0af83a5e...

Current branch diff-target is up to date.
Changes applied before test
commit 263515f3538550213159ce1706d30d4e4c015d7c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Apr 26 13:28:37 2022 +0200

    Add method get_parent_origin()
    
    This will allow the Git loader to incrementally load GitHub forks

See https://jenkins.softwareheritage.org/job/DLDMD/job/tests-on-diff/7/ for more details.

Harbormaster completed remote builds in B28850: Diff 27788.Apr 26 2022, 5:35 PM
vlorentz added a child revision: D7691: Store the result of MetadataFetcher.get_parent_origins.Apr 26 2022, 5:42 PM
vlorentz retitled this revision from Add method get_parent_origin() to Add method get_parent_origins().Apr 27 2022, 1:11 PM
olasd accepted this revision.Apr 27 2022, 1:30 PM

Thanks. Your commit message needs an update :-)

This revision is now accepted and ready to land.Apr 27 2022, 1:30 PM
vlorentz updated this revision to Diff 27817.Apr 27 2022, 1:31 PM

reword commit msg

This revision was landed with ongoing or failed builds.Apr 27 2022, 1:31 PM
Closed by commit rDLDMD01d34fa76821: Add method get_parent_origins() (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDMD01d34fa76821: Add method get_parent_origins().
swh-public-ci added a comment.Apr 27 2022, 1:33 PM

Build is green

Patch application report for D7663 (id=27817)

Rebasing onto 01d34fa768...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-8-D7663.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DLDMD/job/tests-on-diff/8/ for more details.
Harbormaster completed remote builds in B28878: Diff 27817.Apr 27 2022, 1:33 PM
vlorentz added a task: T3273: Use "fork" relationships to speed-up initial load of large repositories.Apr 27 2022, 2:07 PM
Revision Contents
Changeset List
PathSize
swh/
loader/
metadata/
6 lines
17 lines
tests/
data/
https_api.github.com/
324 lines
327 lines
3 lines
33 lines
Diff 27818
View Options
swh/loader/metadata/base.py
Loading...
View Options
swh/loader/metadata/github.py
Loading...
View Options
swh/loader/metadata/tests/data/https_api.github.com/repos_jmarlena_linguist
Loading...
View Options
swh/loader/metadata/tests/data/https_api.github.com/repos_octocat_linguist
Loading...
View Options
swh/loader/metadata/tests/test_base.py
Loading...
View Options
swh/loader/metadata/tests/test_github.py
Loading...
About · Developer information · Legal