Page MenuHomeSoftware Heritage
Differential D8053

Add minimal GitHub metadata mapping
ClosedPublic
Actions

Authored by vlorentz on Jun 29 2022, 6:03 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch
Commits
rDCIDX9085cae01009: Add minimal GitHub metadata mapping
Summary

This introduces the scaffholding for extrinsic metadata mappings

Depends on D8048.

Diff Detail

Event Timeline

vlorentz created this revision.Jun 29 2022, 6:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2022, 6:03 PM
vlorentz added a child revision: D8054: github mapping: Add support for terms outside the codemeta context.Jun 29 2022, 6:08 PM
vlorentz edited the summary of this revision. (Show Details)Jun 29 2022, 6:08 PM
vlorentz added a parent revision: D8048: Move mapping-specific tests to a new directory.
swh-public-ci added a comment.Jun 29 2022, 6:09 PM

Build has FAILED

Patch application report for D8053 (id=29052)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..8fa06ba
Fast-forward
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   48 +-
 swh/indexer/metadata_dictionary/github.py          |   41 +
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  113 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 19 files changed, 1564 insertions(+), 1319 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 8fa06ba290c342c3196b4d58309d1b6c485881b1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/291/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/291/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 29 2022, 6:09 PM
Harbormaster failed remote builds in B30138: Diff 29052!
vlorentz requested review of this revision.Jun 29 2022, 7:58 PM
vlorentz updated this revision to Diff 29062.Jun 30 2022, 11:01 AM

fix cli test

swh-public-ci added a comment.Jun 30 2022, 11:08 AM

Build is green

Patch application report for D8053 (id=29062)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..07074b9
Fast-forward
 swh/indexer/metadata.py                            |   41 +-
 swh/indexer/metadata_detector.py                   |    8 +-
 swh/indexer/metadata_dictionary/__init__.py        |   12 +-
 swh/indexer/metadata_dictionary/base.py            |   48 +-
 swh/indexer/metadata_dictionary/github.py          |   41 +
 swh/indexer/metadata_dictionary/npm.py             |    4 +-
 swh/indexer/metadata_dictionary/ruby.py            |    7 +-
 swh/indexer/tests/metadata_dictionary/__init__.py  |    0
 swh/indexer/tests/metadata_dictionary/test_cff.py  |  220 ++++
 .../tests/metadata_dictionary/test_codemeta.py     |  175 +++
 .../tests/metadata_dictionary/test_github.py       |  113 ++
 .../tests/metadata_dictionary/test_maven.py        |  365 ++++++
 swh/indexer/tests/metadata_dictionary/test_npm.py  |  322 +++++
 .../tests/metadata_dictionary/test_python.py       |  114 ++
 swh/indexer/tests/metadata_dictionary/test_ruby.py |  134 ++
 swh/indexer/tests/test_cli.py                      |    1 +
 swh/indexer/tests/test_metadata.py                 | 1277 --------------------
 swh/indexer/tests/zz_celery/README                 |    2 +
 swh/indexer/tests/zz_celery/__init__.py            |    0
 swh/indexer/tests/{ => zz_celery}/test_tasks.py    |    0
 20 files changed, 1565 insertions(+), 1319 deletions(-)
 create mode 100644 swh/indexer/metadata_dictionary/github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/__init__.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_cff.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_codemeta.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_github.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_maven.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_npm.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_python.py
 create mode 100644 swh/indexer/tests/metadata_dictionary/test_ruby.py
 create mode 100644 swh/indexer/tests/zz_celery/README
 create mode 100644 swh/indexer/tests/zz_celery/__init__.py
 rename swh/indexer/tests/{ => zz_celery}/test_tasks.py (100%)
Changes applied before test
commit 07074b9eec29880698469a623133a94a7122b731
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

commit 244bf36f55fb919f9b9da8503db309a6d816fd30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 13:43:14 2022 +0200

    Move mapping-specific tests to a new directory
    
    We have many of those now; and keeping them all their tests in the same file
    is messy
    
    This causes these tests to run after Celery tests, which breaks them;
    so this commit also renames Celery tests to make them run last.

commit e002b2ee66b305c98a153cc2b57088c179a3fc68
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:19:00 2022 +0200

    Remove given/when/then comments

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/295/ for more details.

Harbormaster completed remote builds in B30148: Diff 29062.Jun 30 2022, 11:08 AM
vlorentz added a task: T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.Jun 30 2022, 3:31 PM
douardda accepted this revision.Jul 4 2022, 11:59 AM
douardda added a subscriber: douardda.

I'm not against this, but I have my doubts on the overall "architecture".

swh/indexer/metadata_dictionary/base.py
51–52

so if I get this right, with this new method, the BaseMapping is getting extended to support both intrinsic and extrinsic MD, right? Why isn't there a similar "intrinsic_metadata_formats()` method then? Not sure I like (or understand?) this "architecture" a lot.

66–67

I find it a bit weird that the SingleFileMapping class is only for intrinsic MD without making this a clear statement by naming the class accordingly.

This revision is now accepted and ready to land.Jul 4 2022, 11:59 AM
vlorentz added inline comments.Jul 4 2022, 12:12 PM
swh/indexer/metadata_dictionary/base.py
51–52

extrinsic metadata is present directly in the journal in RawExtrinsicMetadata objects, each with one of the formats described in this exhaustive list: https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#extrinsic-metadata-formats

intrinsic metadata is present in directories, and is detected by detect_metadata_files, which needs to be smarter, because the set of possible files is unbounded. For example, *.gemspec.

vlorentz added inline comments.Jul 4 2022, 12:16 PM
swh/indexer/metadata_dictionary/base.py
66–67

yeah it's a little messy. I think I'll separate the two class hierarchies (one for extrinsic and one for intrinsic) and make DictMapping and its descendents a mixin; but in a future diff.

douardda added inline comments.Jul 4 2022, 12:29 PM
swh/indexer/metadata_dictionary/base.py
51–52

extrinsic metadata is present directly in the journal in RawExtrinsicMetadata objects, each with one of the formats described in this exhaustive list: https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#extrinsic-metadata-formats

intrinsic metadata is present in directories, and is detected by detect_metadata_files, which needs to be smarter, because the set of possible files is unbounded. For example, *.gemspec.

I guess what I will wait for is a nice user friendly documentation of all this in the end :-)

vlorentz updated this revision to Diff 29122.Jul 4 2022, 12:38 PM

rebase

swh-public-ci added a comment.Jul 4 2022, 12:44 PM

Build is green

Patch application report for D8053 (id=29122)

Rebasing onto 3074268b1b...

Current branch diff-target is up to date.
Changes applied before test
commit 9085cae01009f19a00a9c3b1e56eeb138e4f2775
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 17:53:03 2022 +0200

    Add minimal GitHub metadata mapping
    
    This introduces the scaffholding for extrinsic metadata mappings

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/314/ for more details.

Harbormaster completed remote builds in B30214: Diff 29122.Jul 4 2022, 12:44 PM
Closed by commit rDCIDX9085cae01009: Add minimal GitHub metadata mapping (authored by vlorentz). · Explain WhyJul 4 2022, 12:47 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX9085cae01009: Add minimal GitHub metadata mapping.
vlorentz mentioned this in D8072: Use separate base classes for intrinsic and extrinsic mappings.Jul 4 2022, 1:38 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
indexer/
metadata_dictionary/
12 lines65%
Loading...
22 lines84%
Loading...
41 lines90%
Loading...
tests/
metadata_dictionary/
113 lines100%
Loading...
1 line100%
Loading...

Diff 29062

View Options

swh/indexer/metadata_dictionary/__init__.py

Loading...
View Options

swh/indexer/metadata_dictionary/base.py

Loading...
View Options

swh/indexer/metadata_dictionary/github.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_github.py

Loading...
View Options

swh/indexer/tests/test_cli.py

Loading...
About · Developer information · Legal