Page MenuHomeSoftware Heritage
Differential D8055

github mapping: Add support for more terms from the Codemeta crosswalk
ClosedPublic
Actions

Authored by vlorentz on Jun 30 2022, 10:54 AM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch
Commits
rDCIDX151a3b8a2b69: github mapping: Add support for more terms from the Codemeta crosswalk
Summary

Depends on D8054

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jun 30 2022, 10:54 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 30 2022, 10:54 AM
swh-public-ci added a comment.Jun 30 2022, 11:00 AM

Build has FAILED

Patch application report for D8055 (id=29061)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..445d0ea
Fast-forward
 swh/indexer/codemeta.py                            |    2 +
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   80 +-
 swh/indexer/metadata_dictionary/github.py          |   78 ++
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  126 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 20 files changed, 1647 insertions(+), 1320 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 445d0eaf284d0c46b3487936e1310db12dbd5118
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 30 10:54:03 2022 +0200

    github mapping: Add support for more terms from the Codemeta crosswalk

commit 754c0fba6880c66bb35f43286b70b947e62f2534
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 8fa06ba290c342c3196b4d58309d1b6c485881b1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/294/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/294/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 30 2022, 11:00 AM
Harbormaster failed remote builds in B30147: Diff 29061!
vlorentz edited the summary of this revision. (Show Details)Jun 30 2022, 11:00 AM
vlorentz added a parent revision: D8054: github mapping: Add support for terms outside the codemeta context.
vlorentz updated this revision to Diff 29064.Jun 30 2022, 11:01 AM

fix cli test

swh-public-ci added a comment.Jun 30 2022, 11:08 AM

Build is green

Patch application report for D8055 (id=29064)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..2da2bc3
Fast-forward
 swh/indexer/codemeta.py                            |    2 +
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   80 +-
 swh/indexer/metadata_dictionary/github.py          |   78 ++
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  126 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_cli.py                      |    1 +
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 21 files changed, 1648 insertions(+), 1320 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 2da2bc3ce7b73b775dbba22d3763cea1526a544b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 30 10:54:03 2022 +0200

    github mapping: Add support for more terms from the Codemeta crosswalk

commit a20610b2503ec503e9c0d3b26e809d779c3dbc8e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 07074b9eec29880698469a623133a94a7122b731
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/297/ for more details.

Harbormaster completed remote builds in B30150: Diff 29064.Jun 30 2022, 11:08 AM
vlorentz requested review of this revision.Jun 30 2022, 11:08 AM
vlorentz added a child revision: D8060: Add extrinsic metadata indexer.Jun 30 2022, 3:30 PM
vlorentz added a task: T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.Jun 30 2022, 3:31 PM
douardda accepted this revision.Jul 4 2022, 12:11 PM
This revision is now accepted and ready to land.Jul 4 2022, 12:11 PM
vlorentz updated this revision to Diff 29124.Jul 4 2022, 12:38 PM

rebase

swh-public-ci added a comment.Jul 4 2022, 12:45 PM

Build is green

Patch application report for D8055 (id=29124)

Could not rebase; Attempt merge onto 3074268b1b...

Updating 3074268..151a3b8
Fast-forward
 swh/indexer/codemeta.py                            |   2 +
 swh/indexer/metadata_dictionary/__init__.py        |  12 +-
 swh/indexer/metadata_dictionary/base.py            |  54 ++++++++-
 swh/indexer/metadata_dictionary/github.py          |  78 +++++++++++++
 .../tests/metadata_dictionary/test_github.py       | 126 +++++++++++++++++++++
 swh/indexer/tests/test_cli.py                      |   1 +
 6 files changed, 265 insertions(+), 8 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
Changes applied before test
commit 151a3b8a2b698c999a0efb4f2ee7f5076d8a3076
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jun 30 10:54:03 2022 +0200

    github mapping: Add support for more terms from the Codemeta crosswalk

commit 8948c83972512326bd11eebaf0354b92747a8718
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 18:08:00 2022 +0200

    github mapping: Add support for terms outside the codemeta context

commit 9085cae01009f19a00a9c3b1e56eeb138e4f2775
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/316/ for more details.

Harbormaster completed remote builds in B30216: Diff 29124.Jul 4 2022, 12:45 PM
Closed by commit rDCIDX151a3b8a2b69: github mapping: Add support for more terms from the Codemeta crosswalk (authored by vlorentz). · Explain WhyJul 4 2022, 12:47 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX151a3b8a2b69: github mapping: Add support for more terms from the Codemeta crosswalk.

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
17 lines
tests/
metadata_dictionary/
6 lines

Diff 29129

View Options

swh/indexer/metadata_dictionary/github.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_github.py

Loading...
About · Developer information · Legal