Differential D8046

Remove SingleFileMapping from JsonMapping's base classes
ClosedPublic
Actions

Authored by vlorentz on Jun 29 2022, 12:09 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch

Commits

rDCIDXe6b03db91c99: Remove SingleFileMapping from JsonMapping's base classes

Summary

Extrinsic metadata indexers will not use a 'file' as input,
but will typically use RawExtrinsicMetadata containing formats
in JSON.

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 30119
Build 47068: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 47067: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jun 29 2022, 12:09 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2022, 12:09 PM

Build is green

Patch application report for D8046 (id=29032)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..65edef3
Fast-forward
 swh/indexer/metadata.py                 | 41 ++++++++++++++++++---------------
 swh/indexer/metadata_detector.py        |  8 +++++--
 swh/indexer/metadata_dictionary/base.py | 26 ++++++++++++---------
 swh/indexer/metadata_dictionary/npm.py  |  4 ++--
 swh/indexer/metadata_dictionary/ruby.py |  7 ++++--
 5 files changed, 51 insertions(+), 35 deletions(-)

Changes applied before test

commit 65edef32831949de7b8e14846ecd4fa43bc619ee
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit f7a4bf4e04b3ac4c2fa89cf9b8a5c22e5f0c4d12
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/287/ for more details.

Harbormaster completed remote builds in B30119: Diff 29032.Jun 29 2022, 12:31 PM

vlorentz requested review of this revision.Jun 29 2022, 12:31 PM

ardumont accepted this revision.Jun 29 2022, 3:21 PM

This revision is now accepted and ready to land.Jun 29 2022, 3:21 PM

vlorentz added a task: T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.Jun 30 2022, 3:31 PM

rebase

Build is green

Patch application report for D8046 (id=29114)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..6293230
Fast-forward
 swh/indexer/metadata.py                 | 44 ++++++++++++++++++---------------
 swh/indexer/metadata_detector.py        |  8 ++++--
 swh/indexer/metadata_dictionary/base.py | 26 ++++++++++---------
 swh/indexer/metadata_dictionary/npm.py  |  4 +--
 swh/indexer/metadata_dictionary/ruby.py |  7 ++++--
 5 files changed, 52 insertions(+), 37 deletions(-)

Changes applied before test

commit 62932303c8b8a2a551697f562b0406c03b457712
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit dc035a669329272713b0b977ba2f685501e80ba0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions
    
    This also moves the call to `detect_metadata()` to
    `translate_directory_intrinsic_metadata` so type annotations make more
    sense; and remove a dead/broken code branch in
    `DirectoryMetadataIndexer.index()` that was detected by mypy.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/309/ for more details.

Harbormaster completed remote builds in B30205: Diff 29114.Jul 4 2022, 12:13 PM

rebase

This revision was landed with ongoing or failed builds.Jul 4 2022, 12:29 PM

Closed by commit rDCIDXe6b03db91c99: Remove SingleFileMapping from JsonMapping's base classes (authored by vlorentz). · Explain Why

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDCIDXe6b03db91c99: Remove SingleFileMapping from JsonMapping's base classes.

Build was aborted

Patch application report for D8046 (id=29117)

Could not rebase; Attempt merge onto 1be4e184d4...

Updating 1be4e18..e6b03db
Fast-forward
 swh/indexer/metadata.py                 | 52 +++++++++++++++++----------------
 swh/indexer/metadata_detector.py        |  8 +++--
 swh/indexer/metadata_dictionary/base.py | 26 ++++++++++-------
 swh/indexer/metadata_dictionary/npm.py  |  4 +--
 swh/indexer/metadata_dictionary/ruby.py |  7 +++--
 5 files changed, 55 insertions(+), 42 deletions(-)

Changes applied before test

commit e6b03db91c994fc1d680b76bf0798054a8120b79
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 12:08:49 2022 +0200

    Remove SingleFileMapping from JsonMapping's base classes
    
    Extrinsic metadata indexers will not use a 'file' as input,
    but will typically use RawExtrinsicMetadata containing formats
    in JSON.

commit 8c5a84459d4436af7b4b751e48091a99bce0a7d1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 29 11:01:35 2022 +0200

    Add typing to detect_metadata() and related functions
    
    This also moves the call to `detect_metadata()` to
    `translate_directory_intrinsic_metadata` so type annotations make more
    sense; and remove a dead/broken code branch in
    `DirectoryMetadataIndexer.index()` that was detected by mypy.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/312/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/312/console

Harbormaster failed remote builds in B30208: Diff 29117!Jul 4 2022, 12:32 PM