Page MenuHomeSoftware Heritage

Replace 'normalize' parameter of _translate_dict() with a hook method
ClosedPublic

Authored by vlorentz on Aug 22 2022, 2:26 PM.

Details

Summary

This parameter was only used to execute extra code before
normalize_translation is called. This caused some duplication, and
will not work when switching to a non-JSON-LD internal representation.

Removing it also makes the code of mappings more consistent, by removing
specific field handling from their implementation of the translate
method itself.

Depends on D8274.

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build was aborted

Patch application report for D8275 (id=29889)

Could not rebase; Attempt merge onto 7d7f29fb6a...

Updating 7d7f29f..c8a4571
Fast-forward
 swh/indexer/codemeta.py                     | 23 +++++++++----------
 swh/indexer/metadata_dictionary/base.py     | 28 +++++++++++++-----------
 swh/indexer/metadata_dictionary/cff.py      | 17 ++++++++-------
 swh/indexer/metadata_dictionary/composer.py |  9 ++++----
 swh/indexer/metadata_dictionary/dart.py     | 13 ++++++-----
 swh/indexer/metadata_dictionary/github.py   | 34 +++++++++++++----------------
 swh/indexer/metadata_dictionary/maven.py    | 25 +++++++++++++--------
 swh/indexer/metadata_dictionary/npm.py      | 11 +++++-----
 swh/indexer/metadata_dictionary/nuget.py    |  6 ++---
 swh/indexer/metadata_dictionary/python.py   | 22 ++++++++++---------
 swh/indexer/metadata_dictionary/ruby.py     |  7 +++---
 swh/indexer/namespaces.py                   | 28 ++++++++++++++++++++++++
 12 files changed, 130 insertions(+), 93 deletions(-)
 create mode 100644 swh/indexer/namespaces.py
Changes applied before test
commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/436/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/436/console

Harbormaster returned this revision to the author for changes because remote builds failed.Aug 22 2022, 2:37 PM
Harbormaster failed remote builds in B30942: Diff 29889!

Build is green

Patch application report for D8275 (id=29889)

Could not rebase; Attempt merge onto 7d7f29fb6a...

Updating 7d7f29f..c8a4571
Fast-forward
 swh/indexer/codemeta.py                     | 23 +++++++++----------
 swh/indexer/metadata_dictionary/base.py     | 28 +++++++++++++-----------
 swh/indexer/metadata_dictionary/cff.py      | 17 ++++++++-------
 swh/indexer/metadata_dictionary/composer.py |  9 ++++----
 swh/indexer/metadata_dictionary/dart.py     | 13 ++++++-----
 swh/indexer/metadata_dictionary/github.py   | 34 +++++++++++++----------------
 swh/indexer/metadata_dictionary/maven.py    | 25 +++++++++++++--------
 swh/indexer/metadata_dictionary/npm.py      | 11 +++++-----
 swh/indexer/metadata_dictionary/nuget.py    |  6 ++---
 swh/indexer/metadata_dictionary/python.py   | 22 ++++++++++---------
 swh/indexer/metadata_dictionary/ruby.py     |  7 +++---
 swh/indexer/namespaces.py                   | 28 ++++++++++++++++++++++++
 12 files changed, 130 insertions(+), 93 deletions(-)
 create mode 100644 swh/indexer/namespaces.py
Changes applied before test
commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/441/ for more details.

ardumont added a subscriber: ardumont.

couple of remarks inline.

swh/indexer/metadata_dictionary/maven.py
47

maybe add types?

107–112

types?

This revision is now accepted and ready to land.Aug 22 2022, 4:50 PM

meh, it's Dict[str, Any] and Any everywhere, so not really worth it