HomeSoftware Heritage

loaders: Move the proxy storage filter after the buffer proxy

Description

loaders: Move the proxy storage filter after the buffer proxy

in their pipeline configuration

context: D3976 for some DVCS loaders now send one object at a time to the
storage.

So this will allow batching calls to the *_missing endpoints (for dvcs loaders
e.g. git loader).

This slighly impacts the package loaders but this should tend towards null.

Prior to this we filtered unknown objects and kept a buffer of those unknown
objects to flush to the storage given a threshold hit.

Now, we will buffer all objects and then filter on said buffer of objects. So
we may increase calls to the *_missing endpoints.

Related to D3976
Related to T2373

Details

Provenance
ardumontAuthored on Sep 18 2020, 2:01 PM
ardumontPushed on Sep 18 2020, 3:12 PM
Differential Revision
D3986: loaders: Move the proxy storage filter after the buffer proxy
Parents
rSPSITEc3e76ffac4bb: conf/deposit: Explicit api configuration
Branches
Unknown
Tags
Unknown
Tasks
T2373: git loader OOM when loading huge repository