Differential D4982

ContentPartitionIndexer: Do not index the same content multiple times at once.
ClosedPublic
Actions

Authored by vlorentz on Feb 1 2021, 2:42 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDCIDX4080b9ee931f: ContentPartitionIndexer: Do not index the same content multiple times at once.

Summary

self._index_contents was called multiple times in a loop with the same arguments,
except for the set of hashes to exclude.

It means that, if there were N pages of hashes to exclude, each content was
indexed N times; and the N-1 first iterations didn't even exclude all the
hashes they had to exclude.

Resolves SWH-INDEXER-93 and SWH-INDEXER-7R

(w/ @ardumont)

Diff Detail

Repository

rDCIDX Metadata indexer

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Feb 1 2021, 2:42 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 1 2021, 2:42 PM

Build is green

Patch application report for D4982 (id=17773)

Rebasing onto 3baf8bb919...

Current branch diff-target is up to date.

Changes applied before test

commit 4080b9ee931fe914a91addf1df2d160e56a2d8bb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Feb 1 14:41:23 2021 +0100

    ContentPartitionIndexer: Do not index the same content multiple times at once.
    
    self._index_contents was called multiple times in a loop with the same arguments,
    except for the set of hashes to exclude.
    
    It means that, if there were N pages of hashes to exclude, each content was
    indexed N times; and the N-1 first iterations didn't even exclude all the
    hashes they had to exclude.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/143/ for more details.

Harbormaster completed remote builds in B18914: Diff 17773.Feb 1 2021, 2:45 PM

vlorentz requested review of this revision.Feb 1 2021, 2:45 PM

ardumont accepted this revision.Feb 1 2021, 2:57 PM

This revision is now accepted and ready to land.Feb 1 2021, 2:57 PM

Closed by commit rDCIDX4080b9ee931f: ContentPartitionIndexer: Do not index the same content multiple times at once. (authored by vlorentz). · Explain WhyFeb 1 2021, 3:02 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDCIDX4080b9ee931f: ContentPartitionIndexer: Do not index the same content multiple times at once..