HomeSoftware Heritage

Make the copy process of blob objects run with thread concurrency

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

Make the copy process of blob objects run with thread concurrency

for each batch of messages, dispatch the copy of individual objects in a
ThreadPoolExecutor.
The idea is to allow concurrency to ge beyong process parallelism
provided by kafka consumer groups. Since the copy a one object is mainly
IO bound (check existence in destination objstare, retrieve from source
objstorage, put in destination objstorage) with possibly large delays
(eg. retrieveing a blob from s3 imply a minimum 150/200ms delay before
the HTTP request is replied, whatever the size of the object);
this tries to parallelize those delays.

Details

Provenance
douarddaAuthored on Jan 10 2022, 3:11 PM
douarddaPushed on Jan 21 2022, 9:46 AM
Differential Revision
D6945: Make the copy process of blob objects run with thread concurrency
Build Status
Buildable 26208
Build 40968: test-and-buildJenkins console · Jenkins

Commit No Longer Exists

This commit no longer exists in the repository.