HomeSoftware Heritage

buffer: add a threshold for the estimated size of revision and release batches

Description

buffer: add a threshold for the estimated size of revision and release batches

The size of individual revisions and releases is essentially unbounded.
This means that, when the buffer storage is used as a way of limiting
memory use for an ingestion process, it is still possible to go beyond
the expected memory use when adding a batch of revisions or releases
with large messages or other metadata.

The duration of the database operations for revision_add or release_add is also
commensurate to the size of the objects added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on estimated sizes for batches of revision and
release objects makes this overuse of memory and of database transaction
time much less likely.