Changeset View
Changeset View
Standalone View
Standalone View
swh/storage/tenacious.py
Show First 20 Lines • Show All 45 Lines • ▼ Show 20 Lines | class TenaciousProxyStorage: | ||||
Also provides a error-rate limit feature: if more than n errors occurred during the | Also provides a error-rate limit feature: if more than n errors occurred during the | ||||
insertion of the last p (window_size) objects, stop accepting any insertion. | insertion of the last p (window_size) objects, stop accepting any insertion. | ||||
This proxy is mainly intended to be used in a replayer configuration (aka a | This proxy is mainly intended to be used in a replayer configuration (aka a | ||||
mirror stack), where insertion errors are mostly unexpected (which explains | mirror stack), where insertion errors are mostly unexpected (which explains | ||||
the low default ratio errors/window_size). | the low default ratio errors/window_size). | ||||
Conversely, it should not be used in a loader configuration, as it may | |||||
drop objects without stopping the loader, which leads to holes in the graph. | |||||
Deployments using this proxy should carefully monitor their logs to check any | |||||
failure is expected (because the failed object is corrupted), | |||||
not because of transient errors or issues with the storage backend. | |||||
Sample configuration use case for tenacious storage: | Sample configuration use case for tenacious storage: | ||||
.. code-block:: yaml | .. code-block:: yaml | ||||
storage: | storage: | ||||
cls: tenacious | cls: tenacious | ||||
storage: | storage: | ||||
cls: remote | cls: remote | ||||
▲ Show 20 Lines • Show All 80 Lines • Show Last 20 Lines |