HomeSoftware Heritage

Limit the number of entries in the cache

Description

Limit the number of entries in the cache

The implementation test after each revision if there are
more than 100 000 entries of any kind in the cache. If yes,
it flush the content.
It's quite naive but the memory seems to stay around 5Go
max on big repositories like linux.

This can be still an issue with the current zeromq client because
the origin are sorted by repositories a lot of snapshots of big
repositories can be ingested in parallel.
With the journal implementation, it should be more distributed on
repositories of different sizes

Related to T4313

Details

Provenance
vsellierAuthored on Jun 27 2022, 3:48 PM
vsellierPushed on Jun 28 2022, 10:17 AM
Differential Revision
D8040: Limit the number of entries in the cache
Parents
rDPROV80434e3b2191: Reduce multiplexer logs output
Branches
Unknown
Tags
Unknown
Tasks
T4313: [provenance] some process are oom killed
Build Status
Buildable 30052
Build 46979: test-and-buildJenkins console · Jenkins