Differential D4722

Normalize Codemeta documents by expanding them.
ClosedPublic
Actions

Authored by vlorentz on Dec 11 2020, 1:44 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2876: metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types

Commits

rDSEA407a00e8aa5a: Normalize Codemeta documents by expanding them.

Summary

This causes all fields to be an array of objects, instead of being
a mix of concrete values, arrays, and objects.

ElasticSearch requires this consistency to be able to dynamically
create mappings.

Resolves T2876.

The FIXME will be resolved by D4723.

Diff Detail

Repository

rDSEA Archive search

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 17889
Build 27639: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 27638: arc lint + arc unit

Event Timeline

vlorentz created this revision.Dec 11 2020, 1:44 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 11 2020, 1:44 PM

Build is green

Patch application report for D4722 (id=16724)

Rebasing onto c7cd341796...

Current branch diff-target is up to date.

Changes applied before test

commit 4390bea8dbd89016ba87010d785ed55e16d6038c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Dec 11 13:43:20 2020 +0100

    Normalize Codemeta documents by expanding them.
    
    This causes all fields to be an array of objects, instead of being
    a mix of concrete values, arrays, and objects.
    
    ElasticSearch requires this consistency to be able to dynamically
    create mappings.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/56/ for more details.

Harbormaster completed remote builds in B17889: Diff 16724.Dec 11 2020, 1:46 PM

vlorentz edited the summary of this revision. (Show Details)Dec 11 2020, 1:56 PM

ardumont accepted this revision.Dec 11 2020, 2:07 PM

ardumont added a subscriber: ardumont.

ardumont added inline comments.

swh/search/elasticsearch.py
20	what about: from swh.indexer.codemeta import expand as normalize_intrinsic_metadata ?

This revision is now accepted and ready to land.Dec 11 2020, 2:07 PM

ardumont added inline comments.Dec 11 2020, 2:09 PM

swh/search/elasticsearch.py
32	I kinda found the following nicer to the eyes. intrinsic_metadata = res.get("intrinsic_metadata") if intrinsic_metadata: res["intrinsic_metadata"] = normalize_intrinsic_metadata(intrinsic_metadata) What do you think?

inline function

Build is green

Patch application report for D4722 (id=16726)

Rebasing onto c7cd341796...

Current branch diff-target is up to date.

Changes applied before test

commit 407a00e8aa5a69350689a4a02609bf79a0e75ff3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Dec 11 13:43:20 2020 +0100

    Normalize Codemeta documents by expanding them.
    
    This causes all fields to be an array of objects, instead of being
    a mix of concrete values, arrays, and objects.
    
    ElasticSearch requires this consistency to be able to dynamically
    create mappings.

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/58/ for more details.

Harbormaster completed remote builds in B17891: Diff 16726.Dec 11 2020, 2:13 PM

Closed by commit rDSEA407a00e8aa5a: Normalize Codemeta documents by expanding them. (authored by vlorentz). · Explain WhyDec 11 2020, 2:14 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDSEA407a00e8aa5a: Normalize Codemeta documents by expanding them..