Page MenuHomeSoftware Heritage

Move origin metadata loading from DepositLoader to base PackageLoader.
ClosedPublic

Authored by vlorentz on Fri, Jul 24, 4:52 PM.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Fri, Jul 24, 4:52 PM

Build is green

Patch application report for D3616 (id=12722)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..a3f50f5
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit a3f50f5159462a332c1bc1376c473cefbf58c790
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:52:09 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 7a59b8ef03c3fa29867e41eb29602a8cdff85ee5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/220/ for more details.

I think you messed up the diff, i see only a comment change in the todo ;)

vlorentz updated this revision to Diff 12723.Fri, Jul 24, 5:04 PM

change base commit

Build is green

Patch application report for D3616 (id=12723)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..0e9cd0e
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 183 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 477 insertions(+), 76 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 0e9cd0e472a8f9a2d22d4beed1b2e33a944cdaba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/221/ for more details.

ardumont added inline comments.Mon, Jul 27, 9:37 AM
swh/loader/package/loader.py
447

why do you need that part?

vlorentz added inline comments.Mon, Jul 27, 9:46 AM
swh/loader/package/loader.py
447

What part? (also, the git diff is a bit confusing; I'm going to push another commit that should make it clearer as a side effect)

vlorentz updated this revision to Diff 12736.Mon, Jul 27, 9:49 AM

fix error message

swh/loader/package/loader.py
447

(done)

Build is green

Patch application report for D3616 (id=12736)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..1033f8a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 185 ++++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 ++++++++++++++++++++++
 12 files changed, 478 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 1033f8a7c55a51e834802a77357c09d35cdcae3d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/222/ for more details.

ardumont added inline comments.Mon, Jul 27, 10:26 AM
swh/loader/package/loader.py
447

I saw what seems to be a new try: catch: block around a seemingly unrelated part of your diff ;)

ardumont accepted this revision.Mon, Jul 27, 10:45 AM

Please inline the methods mentioned above

Othewise, i think it's good ;)

\o/

swh/loader/package/loader.py
638

Please inline within the try-catch.
It only adds indirection as it's not reused imo

680

same inline those 2 extra "_create_*" methods here.
It's not reused.

This revision is now accepted and ready to land.Mon, Jul 27, 10:45 AM
ardumont requested changes to this revision.Mon, Jul 27, 10:46 AM

Wrong selection (sorry)

This revision now requires changes to proceed.Mon, Jul 27, 10:46 AM
vlorentz updated this revision to Diff 12740.Mon, Jul 27, 11:49 AM

inline one of the functions; we agreed the other two are find the way they are

Build is green

Patch application report for D3616 (id=12740)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..3cda87a
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  91 +++++------
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 182 +++++++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 184 +++++++++++++++++++++++
 12 files changed, 475 insertions(+), 77 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 3cda87a57e5ef7dea32ab21d9d1b9fd2a1ebc9ec
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:51:49 2020 +0200

    Move origin metadata loading from DepositLoader to base PackageLoader.

commit 0b07f0d4a05ce9d6bd339ed9571ff5d072fce9bc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 16:25:15 2020 +0200

    Add structure ExtrinsicMetadataItem, to replace tuples.
    
    It makes it clearer what it contains.

commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/223/ for more details.

ardumont accepted this revision.Mon, Jul 27, 11:52 AM

indeed

This revision is now accepted and ready to land.Mon, Jul 27, 11:52 AM