HomeSoftware Heritage

buffer: add a threshold for the estimated size of revision and release batches

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

buffer: add a threshold for the estimated size of revision and release batches

The size of individual revisions and releases is essentially unbounded.
This means that, when the buffer storage is used as a way of limiting
memory use for an ingestion process, it is still possible to go beyond
the expected memory use when adding a batch of revisions or releases
with large messages or other metadata.

The duration of the database operations for revision_add or release_add is also
commensurate to the size of the objects added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on estimated sizes for batches of revision and
release objects makes this overuse of memory and of database transaction
time much less likely.

Commit No Longer Exists

This commit no longer exists in the repository.