HomeSoftware Heritage

buffer: add a threshold for the number of revision parents in one batch

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

buffer: add a threshold for the number of revision parents in one batch

The size of individual revisions is essentially unbounded. This means
that, when the buffer storage is used as a way of limiting memory use
for an ingestion process, it is still possible to go beyond the expected
memory use when adding a batch of revisions with extensive histories.

The duration of the database operation for revision_add is also
commensurate to the number of revision parents added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on cumulated number of revision parents per batch
makes this overuse of memory and of database transaction time much less
likely.

Commit No Longer Exists

This commit no longer exists in the repository.