Page MenuHomeSoftware Heritage

Move origin metadata loading from DepositLoader to base PackageLoader.
ClosedPublic

Authored by vlorentz on Jul 24 2020, 4:52 PM.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
package-metadata
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13949
Build 21406: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 21405: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D3616 (id=12722)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..a3f50f5
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit a3f50f5159462a332c1bc1376c473cefbf58c790
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:52:09 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 7a59b8ef03c3fa29867e41eb29602a8cdff85ee5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/220/ for more details.

I think you messed up the diff, i see only a comment change in the todo ;)

Build is green

Patch application report for D3616 (id=12723)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..0e9cd0e
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 0e9cd0e472a8f9a2d22d4beed1b2e33a944cdaba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/221/ for more details.

swh/loader/package/loader.py
447

why do you need that part?

swh/loader/package/loader.py
447

What part? (also, the git diff is a bit confusing; I'm going to push another commit that should make it clearer as a side effect)

fix error message

swh/loader/package/loader.py
447

(done)

Build is green

Patch application report for D3616 (id=12736)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..1033f8a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 185 ++++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 ++++++++++++++++++++++
 12 files changed, 478 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 1033f8a7c55a51e834802a77357c09d35cdcae3d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/222/ for more details.

swh/loader/package/loader.py
447

I saw what seems to be a new try: catch: block around a seemingly unrelated part of your diff ;)

Please inline the methods mentioned above

Othewise, i think it's good ;)

\o/

swh/loader/package/loader.py
636

Please inline within the try-catch.
It only adds indirection as it's not reused imo

682

same inline those 2 extra "_create_*" methods here.
It's not reused.

This revision is now accepted and ready to land.Jul 27 2020, 10:45 AM

Wrong selection (sorry)

This revision now requires changes to proceed.Jul 27 2020, 10:46 AM

inline one of the functions; we agreed the other two are find the way they are

Build is green

Patch application report for D3616 (id=12740)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..3cda87a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 182 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 475 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 3cda87a57e5ef7dea32ab21d9d1b9fd2a1ebc9ec
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/223/ for more details.

This revision is now accepted and ready to land.Jul 27 2020, 11:52 AM