HomeSoftware Heritage

retry: Add retry behavior on pipeline storage with flushing failure

Description

retry: Add retry behavior on pipeline storage with flushing failure

Currently, wrong "hash collisions" are happening a lot on ingestion [1] [2] [3]
The last loading step (flush) is failing on most loaders (git, npm, etc...).

This commits adds the retry behavior to the current pipeline storage deployed.
Which should decrease the frequency of that error.

The remaining hash collision which won't subside should be then real hash
collisions.

[1] https://sentry.softwareheritage.org/share/issue/102aace238fe4ba6b49bcc5531f7c2bf/

[2] https://sentry.softwareheritage.org/share/issue/8e8b48a1d94c465b8109e76311ecdbe7/

[3] https://sentry.softwareheritage.org/share/issue/d4f1208b7eec4b43b11e38494ff039cc/

Details

Provenance
ardumontAuthored on Jan 30 2020, 12:22 PM
ardumontPushed on Jan 30 2020, 1:39 PM
Differential Revision
D2603: pipeline storage: Add retry behavior on flushing failures
Parents
rDSTO1608fcd51f19: Allow deprecated endpoints to be missing from a backend class.
Branches
Unknown
Tags
Unknown
References
tag: v0.0.169
Build Status
Buildable 10373
Build 15430: test-and-buildJenkins