HomeSoftware Heritage

scheduler.backend_es: Leave index opened when streaming bulk

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

scheduler.backend_es: Leave index opened when streaming bulk

Prior to this commit, we had the proper behavior of closing index when done
streaming. Unfortunately, this created too much gc on es nodes down the line.
So for now, we remove that behavior.

Note that this implies we need another cog that makes a pass once in a while on
indices to close.

Also, this has been running on production for 2 weeks now and no more gc issues
arose since then.

Details

Provenance
ardumontAuthored on Feb 3 2020, 9:20 AM
ardumontPushed on Feb 26 2020, 10:45 AM
Differential Revision
D2614: scheduler.backend_es: Leave index opened when streaming bulk
Build Status
Buildable 10757
Build 16150: test-and-buildJenkins

Commit No Longer Exists

This commit no longer exists in the repository.