metadata-search: Return swh-search even when missing from idx_storage.origin_intrinsic_metadata
ClosedPublic
Actions

Authored by vlorentz on Nov 15 2022, 9:31 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T4599: Github descriptions are not used to search origins

Commits

rDWAPPS76c64ea4dc9e: metadata-search: Return swh-search even when missing from idx_storage.

Summary

This is needed, because swh-search may now return results based on extrinsic metadata,
in addition to intrinsic metadata.

I do not want to query idx_storage.origin_extrinsic_metadata here, because it
is not clear how to merge with the existing data structure.

Additionally, I do not think anyone relies on the metadata returned by this
endpoint because it is undocumented and rather inflexible. Instead, I would
like to deprecate returning metadata from this endpoint altogether, as there
is a more appropriate endpoint to get metadata once you have the origin URL.

Resolves T4599.

Diff Detail

Repository

rDWAPPS Web applications

Branch

ext-metadata

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 32798
Build 51389: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 51388: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 15 2022, 9:31 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 15 2022, 9:31 AM

Build is green

Patch application report for D8842 (id=31863)

Rebasing onto ad8558c69d...

Current branch diff-target is up to date.

Changes applied before test

commit 76c64ea4dc9e158bdd5c2730340e04b213999609
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 15 09:31:11 2022 +0100

    metadata-search: Return swh-search even when missing from idx_storage.origin_intrinsic_metadata
    
    This is needed, because swh-search may now return results based on extrinsic metadata,
    in addition to intrinsic metadata.
    
    I do not want to query idx_storage.origin_extrinsic_metadata here, because it
    is not clear how to merge with the existing data structure.
    
    Additionally, I do not think anyone relies on the metadata returned by this
    endpoint because it is undocumented and rather inflexible. Instead, I would
    like to deprecate returning metadata from this endpoint altogether, as there
    is a more appropriate endpoint to get metadata once you have the origin URL.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/2132/ for more details.

Harbormaster completed remote builds in B32798: Diff 31863.Nov 15 2022, 9:40 AM

vlorentz requested review of this revision.Nov 15 2022, 9:40 AM

vlorentz added a child revision: D8843: metadata-search: Skip query to swh-indexer when its results would be discarded.Nov 15 2022, 9:53 AM

ardumont accepted this revision.Nov 15 2022, 11:00 AM

ardumont added a subscriber: ardumont.

ardumont added inline comments.

swh/web/utils/archive.py
372	I gather metadata is a dict with keys all urls present in origin_urls (be there results or not from the indexer storage call line 362), right? (I'm trying to determine whether `metadata[url]` can raise KeyError somehow).