Page MenuHomeSoftware Heritage
Differential D3616

Move origin metadata loading from DepositLoader to base PackageLoader.
ClosedPublic
Actions

Authored by vlorentz on Jul 24 2020, 4:52 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T2496: Write deposit metadata for revisions in the generic metadata storage
Commits
rDLDBASE3cda87a57e5e: Move origin metadata loading from DepositLoader to base PackageLoader.
Summary

Depends on D3615.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
package-metadata
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13949
Build 21406: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 21405: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jul 24 2020, 4:52 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 24 2020, 4:52 PM
swh-public-ci added a comment.Jul 24 2020, 4:54 PM

Build is green

Patch application report for D3616 (id=12722)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..a3f50f5
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit a3f50f5159462a332c1bc1376c473cefbf58c790
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:52:09 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 7a59b8ef03c3fa29867e41eb29602a8cdff85ee5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/220/ for more details.

Harbormaster completed remote builds in B13948: Diff 12722.Jul 24 2020, 4:54 PM
vlorentz mentioned this in D3613: Make the base PackageLoader write extrinsic revision metadata..Jul 24 2020, 4:55 PM
ardumont added a subscriber: ardumont.Jul 24 2020, 5:00 PM

I think you messed up the diff, i see only a comment change in the todo ;)

vlorentz updated this revision to Diff 12723.Jul 24 2020, 5:04 PM

change base commit

swh-public-ci added a comment.Jul 24 2020, 5:07 PM

Build is green

Patch application report for D3616 (id=12723)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..0e9cd0e
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 0e9cd0e472a8f9a2d22d4beed1b2e33a944cdaba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/221/ for more details.

Harbormaster completed remote builds in B13949: Diff 12723.Jul 24 2020, 5:07 PM
ardumont added inline comments.Jul 27 2020, 9:37 AM
swh/loader/package/loader.py
447

why do you need that part?

vlorentz added inline comments.Jul 27 2020, 9:46 AM
swh/loader/package/loader.py
447

What part? (also, the git diff is a bit confusing; I'm going to push another commit that should make it clearer as a side effect)

vlorentz updated this revision to Diff 12736.Jul 27 2020, 9:49 AM

fix error message

swh/loader/package/loader.py
447

(done)

swh-public-ci added a comment.Jul 27 2020, 9:51 AM

Build is green

Patch application report for D3616 (id=12736)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..1033f8a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 185 ++++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 ++++++++++++++++++++++
 12 files changed, 478 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 1033f8a7c55a51e834802a77357c09d35cdcae3d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/222/ for more details.

Harbormaster completed remote builds in B13966: Diff 12736.Jul 27 2020, 9:51 AM
ardumont added inline comments.Jul 27 2020, 10:26 AM
swh/loader/package/loader.py
447

I saw what seems to be a new try: catch: block around a seemingly unrelated part of your diff ;)

ardumont accepted this revision.Jul 27 2020, 10:45 AM

Please inline the methods mentioned above

Othewise, i think it's good ;)

\o/

swh/loader/package/loader.py
636

Please inline within the try-catch.
It only adds indirection as it's not reused imo

682

same inline those 2 extra "_create_*" methods here.
It's not reused.

This revision is now accepted and ready to land.Jul 27 2020, 10:45 AM
ardumont requested changes to this revision.Jul 27 2020, 10:46 AM

Wrong selection (sorry)

This revision now requires changes to proceed.Jul 27 2020, 10:46 AM
vlorentz updated this revision to Diff 12740.Jul 27 2020, 11:49 AM

inline one of the functions; we agreed the other two are find the way they are

swh-public-ci added a comment.Jul 27 2020, 11:51 AM

Build is green

Patch application report for D3616 (id=12740)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..3cda87a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 182 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 475 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 3cda87a57e5ef7dea32ab21d9d1b9fd2a1ebc9ec
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/223/ for more details.

Harbormaster completed remote builds in B13970: Diff 12740.Jul 27 2020, 11:52 AM
ardumont accepted this revision.Jul 27 2020, 11:52 AM

indeed

This revision is now accepted and ready to land.Jul 27 2020, 11:52 AM
Closed by commit rDLDBASE3cda87a57e5e: Move origin metadata loading from DepositLoader to base PackageLoader. (authored by vlorentz). · Explain WhyJul 27 2020, 12:11 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLDBASE3cda87a57e5e: Move origin metadata loading from DepositLoader to base PackageLoader..
vlorentz added a task: T2496: Write deposit metadata for revisions in the generic metadata storage.Jul 29 2020, 7:37 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
package/
deposit/
37 lines95%
Loading...
89 lines93%
Loading...
tests/
38 lines100%
Loading...

Diff 12723

View Options

swh/loader/package/deposit/loader.py

Loading...
View Options

swh/loader/package/loader.py

Loading...
View Options

swh/loader/package/tests/test_loader_metadata.py

Loading...
About · Developer information · Legal