Fix crash when indexing two REMD objects from the same deposit
ClosedPublic
Actions

Authored by vlorentz on Nov 30 2022, 10:13 AM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Maniphest Tasks

T4710: DuplicateId errors when indexing from deposits

Commits

rDCIDXf74b47bcf45d: Fix crash when indexing two REMD objects from the same deposit

Summary

The deduplication code assumed remd.target matches the id of results,
but this is no longer true, as we started using REMD objects whose
origin context was used as result id, when remd.target is a
directory (221d48e242520344c6435c07d444f961637497fe).

Resolves T4710.

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 33045
Build 51792: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 51791: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 30 2022, 10:13 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 30 2022, 10:13 AM

Build is green

Patch application report for D8905 (id=32093)

Could not rebase; Attempt merge onto f44e14b11f...

Updating f44e14b..f74b47b
Fast-forward
 swh/indexer/metadata.py                           |  6 ++--
 swh/indexer/metadata_dictionary/utils.py          |  6 +++-
 swh/indexer/tests/metadata_dictionary/test_npm.py | 11 +++++++
 swh/indexer/tests/test_metadata.py                | 35 ++++++++++++++++++++++-
 4 files changed, 53 insertions(+), 5 deletions(-)

Changes applied before test

commit f74b47bcf45ddf06c441d398511dbcad23af583f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 30 10:11:51 2022 +0100

    Fix crash when indexing two REMD objects from the same deposit
    
    The deduplication code assumed `remd.target` matches the id of results,
    but this is no longer true, as we started using REMD objects whose
    `origin` context was used as result id, when `remd.target` is a
    directory (221d48e242520344c6435c07d444f961637497fe).

commit b2d8afff6fa502173b31431b7129ad89d24b7048
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Nov 30 09:44:21 2022 +0100

    metadata_dictionary: Fix 'Invalid IPv6 URL' crash

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/534/ for more details.

Harbormaster completed remote builds in B33045: Diff 32093.Nov 30 2022, 10:17 AM

vlorentz requested review of this revision.Nov 30 2022, 10:17 AM