Page MenuHomeSoftware Heritage
Differential D8053

Add minimal GitHub metadata mapping
ClosedPublic
Actions

Authored by vlorentz on Jun 29 2022, 6:03 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch
Commits
rDCIDX9085cae01009: Add minimal GitHub metadata mapping
Summary

This introduces the scaffholding for extrinsic metadata mappings

Depends on D8048.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
extrinsic
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30138
Build 47103: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47102: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
5,425 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_cli::test_cli_mapping_list
cli_runner = <click.testing.CliRunner object at 0x7fd18eef2be0> swh_config = '/tmp/pytest-of-jenkins/pytest-0/test_cli_mapping_list0/indexer.yml'
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.merge_values
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.github::swh.indexer.metadata_dictionary.github.GitHubMapping.normalize_license
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.normalize_groupId
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.parse_licenses
View Full Test Results (1 Failed · 349 Passed · 15 Skipped)

Event Timeline

vlorentz created this revision.Jun 29 2022, 6:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2022, 6:03 PM
vlorentz added a child revision: D8054: github mapping: Add support for terms outside the codemeta context.Jun 29 2022, 6:08 PM
vlorentz edited the summary of this revision. (Show Details)Jun 29 2022, 6:08 PM
vlorentz added a parent revision: D8048: Move mapping-specific tests to a new directory.
swh-public-ci added a comment.Jun 29 2022, 6:09 PM

Build has FAILED

Patch application report for D8053 (id=29052)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..8fa06ba
Fast-forward
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   48 +-
 swh/indexer/metadata_dictionary/github.py          |   41 +
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  113 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 19 files changed, 1564 insertions(+), 1319 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 8fa06ba290c342c3196b4d58309d1b6c485881b1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/291/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/291/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 29 2022, 6:09 PM
Harbormaster failed remote builds in B30138: Diff 29052!
vlorentz requested review of this revision.Jun 29 2022, 7:58 PM
vlorentz updated this revision to Diff 29062.Jun 30 2022, 11:01 AM

fix cli test

swh-public-ci added a comment.Jun 30 2022, 11:08 AM

Build is green

Patch application report for D8053 (id=29062)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..07074b9
Fast-forward
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   48 +-
 swh/indexer/metadata_dictionary/github.py          |   41 +
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  113 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_cli.py                      |    1 +
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 20 files changed, 1565 insertions(+), 1319 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 07074b9eec29880698469a623133a94a7122b731
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/295/ for more details.

Harbormaster completed remote builds in B30148: Diff 29062.Jun 30 2022, 11:08 AM
vlorentz added a task: T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.Jun 30 2022, 3:31 PM
douardda accepted this revision.Jul 4 2022, 11:59 AM
douardda added a subscriber: douardda.

I'm not against this, but I have my doubts on the overall "architecture".

swh/indexer/metadata_dictionary/base.py
50–51

so if I get this right, with this new method, the BaseMapping is getting extended to support both intrinsic and extrinsic MD, right? Why isn't there a similar "intrinsic_metadata_formats()` method then? Not sure I like (or understand?) this "architecture" a lot.

66–67

I find it a bit weird that the SingleFileMapping class is only for intrinsic MD without making this a clear statement by naming the class accordingly.

This revision is now accepted and ready to land.Jul 4 2022, 11:59 AM
vlorentz added inline comments.Jul 4 2022, 12:12 PM
swh/indexer/metadata_dictionary/base.py
50–51

extrinsic metadata is present directly in the journal in RawExtrinsicMetadata objects, each with one of the formats described in this exhaustive list: https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#extrinsic-metadata-formats

intrinsic metadata is present in directories, and is detected by detect_metadata_files, which needs to be smarter, because the set of possible files is unbounded. For example, *.gemspec.

vlorentz added inline comments.Jul 4 2022, 12:16 PM
swh/indexer/metadata_dictionary/base.py
66–67

yeah it's a little messy. I think I'll separate the two class hierarchies (one for extrinsic and one for intrinsic) and make DictMapping and its descendents a mixin; but in a future diff.

douardda added inline comments.Jul 4 2022, 12:29 PM
swh/indexer/metadata_dictionary/base.py
50–51

extrinsic metadata is present directly in the journal in RawExtrinsicMetadata objects, each with one of the formats described in this exhaustive list: https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#extrinsic-metadata-formats

intrinsic metadata is present in directories, and is detected by detect_metadata_files, which needs to be smarter, because the set of possible files is unbounded. For example, *.gemspec.

I guess what I will wait for is a nice user friendly documentation of all this in the end :-)

vlorentz updated this revision to Diff 29122.Jul 4 2022, 12:38 PM

rebase

swh-public-ci added a comment.Jul 4 2022, 12:44 PM

Build is green

Patch application report for D8053 (id=29122)

Rebasing onto 3074268b1b...

Current branch diff-target is up to date.
Changes applied before test
commit 9085cae01009f19a00a9c3b1e56eeb138e4f2775
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/314/ for more details.

Harbormaster completed remote builds in B30214: Diff 29122.Jul 4 2022, 12:44 PM
Closed by commit rDCIDX9085cae01009: Add minimal GitHub metadata mapping (authored by vlorentz). · Explain WhyJul 4 2022, 12:47 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX9085cae01009: Add minimal GitHub metadata mapping.
vlorentz mentioned this in D8072: Use separate base classes for intrinsic and extrinsic mappings.Jul 4 2022, 1:38 PM

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
12 lines
22 lines
41 lines
tests/
metadata_dictionary/
113 lines

Diff 29052

View Options

swh/indexer/metadata_dictionary/__init__.py

Loading...
View Options

swh/indexer/metadata_dictionary/base.py

Loading...
View Options

swh/indexer/metadata_dictionary/github.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_github.py

Loading...
About · Developer information · Legal