HomeSoftware Heritage

scheduler.backend_es: Leave index opened when streaming bulk

Description

scheduler.backend_es: Leave index opened when streaming bulk

Prior to this commit, we had the proper behavior of closing index when done
streaming. Unfortunately, this created too much gc on es nodes down the line.
So for now, we remove that behavior.

Note that this implies we need another cog that makes a pass once in a while on
indices to close.

Also, this has been running on production for 2 weeks now and no more gc issues
arose since then.

Details

Provenance
ardumontAuthored on Feb 3 2020, 9:20 AM
ardumontPushed on Feb 26 2020, 10:45 AM
Differential Revision
D2614: scheduler.backend_es: Leave index opened when streaming bulk
Parents
rDSCHaf58466e6ec3: backend: Make create_task_type idempotent
Branches
Unknown
Tags
Unknown
Build Status
Buildable 10757
Build 16150: test-and-buildJenkins