Page MenuHomeSoftware Heritage
Differential D8054

github mapping: Add support for terms outside the codemeta context
ClosedPublic
Actions

Authored by vlorentz on Jun 29 2022, 6:08 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch
Commits
rDCIDX8948c8397251: github mapping: Add support for terms outside the codemeta context
Summary

Depends on D8053.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
extrinsic
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30140
Build 47107: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47106: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
5,519 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_mapping_list
cli_runner = <click.testing.CliRunner object at 0x7f832844e2b0> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_mapping_list0/indexer.yml'
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.merge_values
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.normalize_license
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.translate_forks_count
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.normalize_groupId
View Full Test Results (1 Failed · 350 Passed · 15 Skipped)

Event Timeline

vlorentz created this revision.Jun 29 2022, 6:08 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2022, 6:08 PM
swh-public-ci added a comment.Jun 29 2022, 6:15 PM

Build has FAILED

Patch application report for D8054 (id=29054)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..754c0fb
Fast-forward
 swh/indexer/codemeta.py                            |    2 +
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   80 +-
 swh/indexer/metadata_dictionary/github.py          |   73 ++
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  122 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 20 files changed, 1638 insertions(+), 1320 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 754c0fba6880c66bb35f43286b70b947e62f2534
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 8fa06ba290c342c3196b4d58309d1b6c485881b1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/292/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/292/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 29 2022, 6:15 PM
Harbormaster failed remote builds in B30140: Diff 29054!
vlorentz requested review of this revision.Jun 29 2022, 7:59 PM
vlorentz added a child revision: D8055: github mapping: Add support for more terms from the Codemeta crosswalk.Jun 30 2022, 11:00 AM
vlorentz updated this revision to Diff 29063.Jun 30 2022, 11:01 AM

fix cli test

swh-public-ci added a comment.Jun 30 2022, 11:08 AM

Build is green

Patch application report for D8054 (id=29063)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..a20610b
Fast-forward
 swh/indexer/codemeta.py                            |    2 +
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   80 +-
 swh/indexer/metadata_dictionary/github.py          |   73 ++
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  122 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_cli.py                      |    1 +
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 21 files changed, 1639 insertions(+), 1320 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit a20610b2503ec503e9c0d3b26e809d779c3dbc8e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 07074b9eec29880698469a623133a94a7122b731
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/296/ for more details.

Harbormaster completed remote builds in B30149: Diff 29063.Jun 30 2022, 11:08 AM
vlorentz added a task: T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.Jun 30 2022, 3:31 PM
douardda accepted this revision.Jul 4 2022, 12:06 PM
douardda added a subscriber: douardda.

ok but the wording of the mark_production decorator

swh/indexer/metadata_dictionary/base.py
27

not sure I find the name mark_productions easy to grasp.

Would'nt something like produce_terms be more explicit/readable?

eg in:

@mark_productions(FORGEFED_URI, ["forks"])
@mark_productions(ACTIVITYSTREAMS_URI, ["totalItems"])
def translate_forks_count(
    self, translated_metadata: Dict[str, Any], v: Any
) -> None:
   ...

would give:

@produce_terms(FORGEFED_URI, ["forks"])
@produce_terms(ACTIVITYSTREAMS_URI, ["totalItems"])
def translate_forks_count(
    self, translated_metadata: Dict[str, Any], v: Any
) -> None:
   ...
This revision is now accepted and ready to land.Jul 4 2022, 12:06 PM
vlorentz updated this revision to Diff 29123.Jul 4 2022, 12:38 PM

rebase + rename decorator

swh-public-ci added a comment.Jul 4 2022, 12:45 PM

Build is green

Patch application report for D8054 (id=29123)

Could not rebase; Attempt merge onto 3074268b1b...

Updating 3074268..8948c83
Fast-forward
 swh/indexer/codemeta.py                            |   2 +
 swh/indexer/metadata_dictionary/__init__.py        |  12 +-
 swh/indexer/metadata_dictionary/base.py            |  54 ++++++++-
 swh/indexer/metadata_dictionary/github.py          |  73 ++++++++++++
 .../tests/metadata_dictionary/test_github.py       | 122 +++++++++++++++++++++
 swh/indexer/tests/test_cli.py                      |   1 +
 6 files changed, 256 insertions(+), 8 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
Changes applied before test
commit 8948c83972512326bd11eebaf0354b92747a8718
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 9085cae01009f19a00a9c3b1e56eeb138e4f2775
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/315/ for more details.

Harbormaster completed remote builds in B30215: Diff 29123.Jul 4 2022, 12:45 PM
Closed by commit rDCIDX8948c8397251: github mapping: Add support for terms outside the codemeta context (authored by vlorentz). · Explain WhyJul 4 2022, 12:47 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX8948c8397251: github mapping: Add support for terms outside the codemeta context.

Revision Contents
Changeset List

PathSize
swh/
indexer/
2 lines
metadata_dictionary/
36 lines
38 lines
tests/
metadata_dictionary/
13 lines

Diff 29054

View Options

swh/indexer/codemeta.py

Loading...
View Options

swh/indexer/metadata_dictionary/base.py

Loading...
View Options

swh/indexer/metadata_dictionary/github.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_github.py

Loading...
About · Developer information · Legal