HomeSoftware Heritage

orchestrator: Send contents batch on a per indexer configuration basis

Description

orchestrator: Send contents batch on a per indexer configuration basis

Prior to this, each indexer received the same batch of contents as the
orchestrator.

As some of our indexer need quite some time to finish when the batch
size is huge enough, the only reasonable step was to reduce such batch
size. But, in effect, this impacted as well other indexers which did
not need such restriction (this for example creates more little db
transations).

The introduction of such option avoids that impact.

This also introduces the option to check the presence on a per indexer
basis.

Related T713

Details

Provenance
ardumontAuthored on May 17 2017, 1:42 PM
ardumontPushed on May 17 2017, 2:14 PM
Parents
rDCIDXcedc5112202e: rehash: do something slighly more sensible with exceptions
Branches
Unknown
Tags
Unknown