Page MenuHomeSoftware Heritage

loaders: Move the proxy storage filter after the buffer proxy
ClosedPublic

Authored by ardumont on Sep 18 2020, 2:44 PM.

Details

Summary

in their pipeline configuration

context: D3976 for some DVCS loaders now send one object at a time to the
storage.

So this will allow batching calls to the *_missing endpoints (for dvcs loaders
e.g. git loader).

This slighly impacts the package loaders but this should tend towards null.

Prior to this we filtered unknown objects and kept a buffer of those unknown
objects to flush to the storage given a threshold hit.

Now, we will buffer all objects and then filter on said buffer of objects. So
we may increase calls to the *_missing endpoints.

Related to D3976
Related to D3986
Related to T2373

Test Plan

docker rebuild which include the latest loader-git 0.11.0 and this setup patch.

time doco exec swh-loader swh --log-level DEBUG loader run git git://git.savannah.gnu.org/guix

loader does its job without a gazillion small calls to the *_missing endpoints.

Diff Detail

Repository
rDENV Development environment
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 15333
Build 23615: arc lint + arc unit