Page MenuHomeSoftware Heritage
Differential D8282

Add class attribute 'uri_fields' to deduplicate simply URI fields
ClosedPublic
Actions

Authored by vlorentz on Aug 22 2022, 4:03 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDCIDX39bb4211697b: Add class attribute 'uri_fields' to deduplicate simply URI fields
Summary

Depends on D8279.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
rdflib
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30956
Build 48419: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 48418: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_codemeta::test_crosstable
def test_crosstable(): > assert {k: str(v) for (k, v) in CROSSWALK_TABLE["NodeJS"].items()} == { "repository": "http://schema.org/codeRepository",
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.make_absolute_uri
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.normalize_license
9 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.translate_forks_count
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.translate_stargazers_count
View Full Test Results (1 Failed · 338 Passed · 11 Skipped)

Event Timeline

vlorentz created this revision.Aug 22 2022, 4:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 22 2022, 4:03 PM
vlorentz updated this revision to Diff 29905.Aug 22 2022, 4:07 PM

rebase

swh-public-ci added a comment.Aug 22 2022, 4:11 PM

Build has FAILED

Patch application report for D8282 (id=29903)

Could not rebase; Attempt merge onto 7d7f29fb6a...

Updating 7d7f29f..416543b
Fast-forward
 docs/metadata-workflow.rst                         |   2 +-
 mypy.ini                                           |   3 +
 requirements.txt                                   |   1 +
 swh/indexer/codemeta.py                            |  64 +++------
 swh/indexer/metadata_dictionary/base.py            | 146 +++++++++++++++++----
 swh/indexer/metadata_dictionary/cff.py             |  82 +++++++-----
 swh/indexer/metadata_dictionary/composer.py        |  41 +++---
 swh/indexer/metadata_dictionary/dart.py            |  47 +++----
 swh/indexer/metadata_dictionary/github.py          | 135 +++++++++----------
 swh/indexer/metadata_dictionary/maven.py           |  99 +++++++-------
 swh/indexer/metadata_dictionary/npm.py             | 141 +++++++++++++-------
 swh/indexer/metadata_dictionary/nuget.py           |  76 +++++------
 swh/indexer/metadata_dictionary/python.py          |  44 ++++---
 swh/indexer/metadata_dictionary/ruby.py            |  47 +++----
 swh/indexer/metadata_dictionary/utils.py           |  72 ++++++++++
 swh/indexer/namespaces.py                          |  12 ++
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  11 +-
 .../tests/metadata_dictionary/test_composer.py     |   7 +-
 swh/indexer/tests/metadata_dictionary/test_dart.py |  17 ++-
 .../tests/metadata_dictionary/test_github.py       |   2 +-
 .../tests/metadata_dictionary/test_maven.py        |  32 ++---
 swh/indexer/tests/metadata_dictionary/test_npm.py  |   7 +-
 .../tests/metadata_dictionary/test_nuget.py        |  53 ++++----
 .../tests/metadata_dictionary/test_python.py       |   9 +-
 swh/indexer/tests/metadata_dictionary/test_ruby.py |   2 +
 swh/indexer/tests/test_codemeta.py                 |  32 +----
 swh/indexer/tests/test_origin_metadata.py          |   6 +-
 swh/indexer/tests/utils.py                         |  13 --
 28 files changed, 668 insertions(+), 535 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/utils.py
 create mode 100644 swh/indexer/namespaces.py
Changes applied before test
commit 416543b631693d37328e2f7d068df54729858914
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 16:03:05 2022 +0200

    Add class attribute 'uri_fields' to deduplicate simply URI fields

commit d9732bd541099164a1ac0bba4e176270a631c172
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 14:20:20 2022 +0200

    Refactor metadata mappings using rdflib.Graph instead of JSON-LD internally
    
    Motivation:
    
    1. It makes it easier to visualize what is actually happening when modifying
       the graph, by working explicitly on triples instead of a JSON-LD (a tree
       serialization of the graph).
    
    2. Remove the need for the hacky `merge_values()` function (and possibly
       `merge_documents()` in a future commit)
    
    3. It also catches malformed data exactly where it is added in the document
       (the call to rdflib.Graph.add()) instead of at the end of the mapping
       when running compaction/expansion.
    
    Downsides:
    
    1. Tests are clunkier, because they relied on deterministic order of
       unordered lists; but rdflib does not guarantee it
    
    2. Code is longer
    
    3. Extra dependency (which we will need at some point if we want to
       import from RDF datasets, anyway)

commit b9f206bfe0a8e2592708e7f1728643654564f32a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 15:47:49 2022 +0200

    Remove 'keywords' from test files
    
    Their order is nondeterministic, it just happens to work with
    the way we use PyLD.

commit d5207e9521d982d2c170399b4855de3b6d9e8005
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:20:14 2022 +0200

    Add base XmlMapping to deduplicate between MavenMapping and NugetMapping

commit c4cd68f6ae0ea92be6e312a37fcf0fe597e7616f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:18:39 2022 +0200

    nuget: Remove test-specific code from the main class

commit b09e2bcfc73d72a670dbedfe5f8334d0036ce195
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:14:05 2022 +0200

    nuget: Inherit directly from BaseIntrinsicMapping

commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/447/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/447/console

Harbormaster failed remote builds in B30956: Diff 29903!Aug 22 2022, 4:11 PM
swh-public-ci added a comment.Aug 22 2022, 4:15 PM

Build is green

Patch application report for D8282 (id=29905)

Could not rebase; Attempt merge onto 7d7f29fb6a...

Updating 7d7f29f..42e663d
Fast-forward
 docs/metadata-workflow.rst                         |   2 +-
 mypy.ini                                           |   3 +
 requirements.txt                                   |   1 +
 swh/indexer/codemeta.py                            |  66 +++-------
 swh/indexer/metadata_dictionary/base.py            | 146 +++++++++++++++++----
 swh/indexer/metadata_dictionary/cff.py             |  82 +++++++-----
 swh/indexer/metadata_dictionary/composer.py        |  41 +++---
 swh/indexer/metadata_dictionary/dart.py            |  47 +++----
 swh/indexer/metadata_dictionary/github.py          | 135 +++++++++----------
 swh/indexer/metadata_dictionary/maven.py           |  99 +++++++-------
 swh/indexer/metadata_dictionary/npm.py             | 141 +++++++++++++-------
 swh/indexer/metadata_dictionary/nuget.py           |  76 +++++------
 swh/indexer/metadata_dictionary/python.py          |  44 ++++---
 swh/indexer/metadata_dictionary/ruby.py            |  47 +++----
 swh/indexer/metadata_dictionary/utils.py           |  72 ++++++++++
 swh/indexer/namespaces.py                          |  12 ++
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  11 +-
 .../tests/metadata_dictionary/test_composer.py     |   7 +-
 swh/indexer/tests/metadata_dictionary/test_dart.py |  17 ++-
 .../tests/metadata_dictionary/test_github.py       |   2 +-
 .../tests/metadata_dictionary/test_maven.py        |  32 ++---
 swh/indexer/tests/metadata_dictionary/test_npm.py  |   7 +-
 .../tests/metadata_dictionary/test_nuget.py        |  53 ++++----
 .../tests/metadata_dictionary/test_python.py       |   9 +-
 swh/indexer/tests/metadata_dictionary/test_ruby.py |   2 +
 swh/indexer/tests/test_codemeta.py                 |  32 +----
 swh/indexer/tests/test_origin_metadata.py          |   6 +-
 swh/indexer/tests/utils.py                         |  13 --
 28 files changed, 669 insertions(+), 536 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/utils.py
 create mode 100644 swh/indexer/namespaces.py
Changes applied before test
commit 42e663d21818e7a2df80f5fe2d1cfd22ec90db6a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 16:03:05 2022 +0200

    Add class attribute 'uri_fields' to deduplicate simply URI fields

commit 885094479a97fd37ec624f781c4c2f87fc87a1f2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 14:20:20 2022 +0200

    Refactor metadata mappings using rdflib.Graph instead of JSON-LD internally
    
    Motivation:
    
    1. It makes it easier to visualize what is actually happening when modifying
       the graph, by working explicitly on triples instead of a JSON-LD (a tree
       serialization of the graph).
    
    2. Remove the need for the hacky `merge_values()` function (and possibly
       `merge_documents()` in a future commit)
    
    3. It also catches malformed data exactly where it is added in the document
       (the call to rdflib.Graph.add()) instead of at the end of the mapping
       when running compaction/expansion.
    
    Downsides:
    
    1. Tests are clunkier, because they relied on deterministic order of
       unordered lists; but rdflib does not guarantee it
    
    2. Code is longer
    
    3. Extra dependency (which we will need at some point if we want to
       import from RDF datasets, anyway)

commit b9f206bfe0a8e2592708e7f1728643654564f32a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 15:47:49 2022 +0200

    Remove 'keywords' from test files
    
    Their order is nondeterministic, it just happens to work with
    the way we use PyLD.

commit d5207e9521d982d2c170399b4855de3b6d9e8005
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:20:14 2022 +0200

    Add base XmlMapping to deduplicate between MavenMapping and NugetMapping

commit c4cd68f6ae0ea92be6e312a37fcf0fe597e7616f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:18:39 2022 +0200

    nuget: Remove test-specific code from the main class

commit b09e2bcfc73d72a670dbedfe5f8334d0036ce195
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 13:14:05 2022 +0200

    nuget: Inherit directly from BaseIntrinsicMapping

commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/449/ for more details.

Harbormaster completed remote builds in B30958: Diff 29905.Aug 22 2022, 4:15 PM
vlorentz requested review of this revision.Aug 22 2022, 4:15 PM
ardumont accepted this revision.Aug 23 2022, 10:10 AM
This revision is now accepted and ready to land.Aug 23 2022, 10:10 AM
vlorentz updated this revision to Diff 29924.Aug 23 2022, 10:52 AM

rebase

swh-public-ci added a comment.Aug 23 2022, 11:03 AM

Build was aborted

Patch application report for D8282 (id=29924)

Could not rebase; Attempt merge onto d5207e9521...

Updating d5207e9..39bb421
Fast-forward
 docs/metadata-workflow.rst                         |   2 +-
 mypy.ini                                           |   3 +
 requirements.txt                                   |   1 +
 swh/indexer/codemeta.py                            |  51 ++------
 swh/indexer/metadata_dictionary/base.py            |  98 +++++++++++----
 swh/indexer/metadata_dictionary/cff.py             |  81 ++++++------
 swh/indexer/metadata_dictionary/composer.py        |  40 +++---
 swh/indexer/metadata_dictionary/dart.py            |  40 +++---
 swh/indexer/metadata_dictionary/github.py          | 119 ++++++++----------
 swh/indexer/metadata_dictionary/maven.py           |  72 ++++++-----
 swh/indexer/metadata_dictionary/npm.py             | 138 +++++++++++++--------
 swh/indexer/metadata_dictionary/nuget.py           |  53 ++++----
 swh/indexer/metadata_dictionary/python.py          |  40 +++---
 swh/indexer/metadata_dictionary/ruby.py            |  44 +++----
 swh/indexer/metadata_dictionary/utils.py           |  72 +++++++++++
 swh/indexer/namespaces.py                          |  20 +--
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  11 +-
 .../tests/metadata_dictionary/test_composer.py     |   7 +-
 swh/indexer/tests/metadata_dictionary/test_dart.py |  17 +--
 .../tests/metadata_dictionary/test_github.py       |   2 +-
 .../tests/metadata_dictionary/test_maven.py        |  32 ++---
 swh/indexer/tests/metadata_dictionary/test_npm.py  |   7 +-
 .../tests/metadata_dictionary/test_nuget.py        |  41 +++---
 .../tests/metadata_dictionary/test_python.py       |   9 +-
 swh/indexer/tests/metadata_dictionary/test_ruby.py |   2 +
 swh/indexer/tests/test_codemeta.py                 |  32 +----
 swh/indexer/tests/test_origin_metadata.py          |   6 +-
 swh/indexer/tests/utils.py                         |  13 --
 28 files changed, 573 insertions(+), 480 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/utils.py
Changes applied before test
commit 39bb4211697b8890148e4c3b719f612ac7673bdf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 16:03:05 2022 +0200

    Add class attribute 'uri_fields' to deduplicate simply URI fields

commit f72d095f425224f16be1bc564f5cc4ed709fb47a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 14:20:20 2022 +0200

    Refactor metadata mappings using rdflib.Graph instead of JSON-LD internally
    
    Motivation:
    
    1. It makes it easier to visualize what is actually happening when modifying
       the graph, by working explicitly on triples instead of a JSON-LD (a tree
       serialization of the graph).
    
    2. Remove the need for the hacky `merge_values()` function (and possibly
       `merge_documents()` in a future commit)
    
    3. It also catches malformed data exactly where it is added in the document
       (the call to rdflib.Graph.add()) instead of at the end of the mapping
       when running compaction/expansion.
    
    Downsides:
    
    1. Tests are clunkier, because they relied on deterministic order of
       unordered lists; but rdflib does not guarantee it
    
    2. Code is longer
    
    3. Extra dependency (which we will need at some point if we want to
       import from RDF datasets, anyway)

commit 97f5fdcdcc3ac76d2b4680dbcd4f2b5d4c557293
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 22 15:47:49 2022 +0200

    Remove 'keywords' from test files
    
    Their order is nondeterministic, it just happens to work with
    the way we use PyLD.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/457/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/457/console

Harbormaster failed remote builds in B30976: Diff 29924!Aug 23 2022, 11:03 AM
This revision was landed with ongoing or failed builds.Aug 23 2022, 11:28 AM
Closed by commit rDCIDX39bb4211697b: Add class attribute 'uri_fields' to deduplicate simply URI fields (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX39bb4211697b: Add class attribute 'uri_fields' to deduplicate simply URI fields.

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
13 lines
5 lines
6 lines
6 lines
12 lines
11 lines
5 lines

Diff 29903

View Options

swh/indexer/metadata_dictionary/base.py

Loading...
View Options

swh/indexer/metadata_dictionary/cff.py

Loading...
View Options

swh/indexer/metadata_dictionary/composer.py

Loading...
View Options

swh/indexer/metadata_dictionary/dart.py

Loading...
View Options

swh/indexer/metadata_dictionary/npm.py

Loading...
View Options

swh/indexer/metadata_dictionary/nuget.py

Loading...
View Options

swh/indexer/metadata_dictionary/ruby.py

Loading...
About · Developer information · Legal