HomeSoftware Heritage

backfill: only flush the journal writer on every batch

Description

backfill: only flush the journal writer on every batch

This module's use of write_addition predated the introduction of reliable
writing in swh.journal; Since this introduction, the backfiller has been
flushing the kafka writer after writing each single object, leading to a 3x
measured slowdown on backfilling contents.

Details

Provenance
olasdAuthored on Nov 12 2020, 6:04 PM
olasdPushed on Nov 13 2020, 11:55 AM
Differential Revision
D4471: backfill: only flush the journal writer on every batch
Parents
rDSTO248a04b5b8a2: Don't use string expansions in debug logging
Branches
Unknown
Tags
Unknown
References
tag: v0.17.2
Build Status
Buildable 17078
Build 26359: test-and-buildJenkins console · Jenkins