Page MenuHomeSoftware Heritage

deposit loader: Write revision metadata to the metadata storage.
ClosedPublic

Authored by vlorentz on Jul 24 2020, 2:38 PM.

Details

Summary

In addition to writing them in the 'metadata' field of revision objects,
for now.

Depends on D3613.

Event Timeline

Build is green

Patch application report for D3614 (id=12717)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..1645cf8
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  48 +++---
 swh/loader/package/deposit/tests/test_deposit.py |  46 +++++-
 swh/loader/package/loader.py                     | 130 ++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 202 +++++++++++++++++++++++
 12 files changed, 423 insertions(+), 52 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 1645cf8bf4fe7c5c811dfcd98934b20798bab24d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 24f8137107c558ad4c5849d36f153782758a0cca
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/215/ for more details.

Looks good to me.

swh/loader/package/deposit/loader.py
83

why is the discovery-date None?

(do you plan to retrieve it later?)


nvm saw that it fallbacks to the visit in the package loader now ;)

This revision is now accepted and ready to land.Jul 24 2020, 4:03 PM

Build is green

Patch application report for D3614 (id=12720)

Could not rebase; Attempt merge onto 14f700d0fa...

Updating 14f700d..7d255ac
Fast-forward
 swh/loader/package/archive/loader.py             |  10 +-
 swh/loader/package/archive/tests/test_archive.py |   4 +-
 swh/loader/package/cran/loader.py                |   6 +-
 swh/loader/package/debian/loader.py              |   6 +-
 swh/loader/package/deposit/loader.py             |  48 ++++---
 swh/loader/package/deposit/tests/test_deposit.py |  46 ++++++-
 swh/loader/package/loader.py                     | 124 +++++++++++++++++-
 swh/loader/package/nixguix/loader.py             |   6 +-
 swh/loader/package/npm/loader.py                 |  10 +-
 swh/loader/package/pypi/loader.py                |   6 +-
 swh/loader/package/tests/test_loader.py          |   1 -
 swh/loader/package/tests/test_loader_metadata.py | 158 +++++++++++++++++++++++
 12 files changed, 375 insertions(+), 50 deletions(-)
 create mode 100644 swh/loader/package/tests/test_loader_metadata.py
Changes applied before test
commit 7d255ac66701b5f7b21a84dd6af21dcbe98b9a25
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:38:25 2020 +0200

    deposit loader: Write revision metadata to the metadata storage.
    
    In addition to writing them in the 'metadata' field of revision objects,
    for now.

commit 8851951fc1689b402f44f087dad49d8ffc62d15e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:04:04 2020 +0200

    Make the base PackageLoader write extrinsic revision metadata.
    
    For now, no loader outside the tests use it; but DepositLoader will soon.

commit ec522be394ebe8eefd489a85e21a3d7ac593851d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jul 24 14:00:17 2020 +0200

    Rename PackageInfo.raw to PackageInfo.raw_info and remove it from BasePackageInfo.
    
    The rename is to disambiguate with 'raw metadata', which may differ from the
    raw info.
    And the base PackageLoader doesn't need to access this field, so removing
    it from BasePackageInfo.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/218/ for more details.