HomeSoftware Heritage

loaders: Move the proxy storage filter after the buffer proxy

This commit no longer exists in the repository. It may have been part of a branch which was deleted.

Description

loaders: Move the proxy storage filter after the buffer proxy

in their pipeline configuration

context: D3976 for some DVCS loaders now send one object at a time to the
storage.

So this will allow batching calls to the *_missing endpoints (for dvcs loaders
e.g. git loader).

This slighly impacts the package loaders but this should tend towards null.

Prior to this we filtered unknown objects and kept a buffer of those unknown
objects to flush to the storage given a threshold hit.

Now, we will buffer all objects and then filter on said buffer of objects. So
we may increase calls to the *_missing endpoints.

Related to D3976
Related to T2373

Details

Provenance
ardumontAuthored on Sep 18 2020, 2:01 PM
ardumontPushed on Sep 18 2020, 3:12 PM
Differential Revision
D3986: loaders: Move the proxy storage filter after the buffer proxy
Tasks
T2373: git loader OOM when loading huge repository

Commit No Longer Exists

This commit no longer exists in the repository.