Page MenuHomeSoftware Heritage
Create Task
Maniphest T2862

vault: retry RPC calls on transient network errors
Closed, MigratedEdits Locked
Actions

Description

Until now, vault cooking fails on any network errors.
A retrying mechanism must be used on RPC calls to storage and scheduler to be resilient in case of transient errors during those calls.

Related to T1191

Revisions and Commits

rDVAU Software Heritage Vault
AbandonedD4630 WIP backend: add retrying on IO error with remote services
rSPSITE puppet-swh-site
D4941rSPSITEd17fc97d35fd cooker: Add retry behavior on storage instance
D4677rSPSITE91aaf9256939 vault: Add retry ability to the storage

Related Objects

Event Timeline

tenma changed the task status from Open to Work in Progress.Dec 7 2020, 5:35 PM
tenma triaged this task as Normal priority.
tenma created this task.
ardumont added a revision: D4677: vault: Add retry ability to the storage.Dec 7 2020, 5:47 PM
ardumont added a revision: D4630: WIP backend: add retrying on IO error with remote services.Dec 7 2020, 6:39 PM
ardumont added a commit: rSPSITE91aaf9256939: vault: Add retry ability to the storage.Dec 8 2020, 5:22 PM
ardumont added a subscriber: ardumont.Dec 8 2020, 5:57 PM

Vault 0.5.0 packaged (includes a fix from @tenma about dropping the unused default configuration).
Vault configuration adapted to add the retry behavior puppet side.

This is deployed on staging and, from my tryouts, it seems to behave fine.
Remains to deploy it to the vault server now (vangogh.euwest.azure).

tenma added a revision: D4941: cooker: Add retry behavior on storage instance.Jan 25 2021, 3:43 PM
ardumont added a commit: rSPSITEd17fc97d35fd: cooker: Add retry behavior on storage instance.Jan 25 2021, 4:00 PM
ardumont added a comment.EditedJan 25 2021, 5:11 PM

Deployed configuration change in staging without issues.
Deployed configuration change in produciton as well.

vlorentz closed this task as Resolved.Sep 10 2021, 11:28 AM
vlorentz claimed this task.
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:32 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: vlorentz, gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal