Page MenuHomeSoftware Heritage
Create Task
Maniphest T4427

Deal with recurring Sentry reports of temporary server shutdowns
Closed, MigratedEdits Locked
Actions

Description

Sentry has tons of reports from RPC clients that are caused by their servers being temporarily down (typically, a postgresql database).

For example: https://sentry.softwareheritage.org/share/issue/42e620d6e6194d9e985fe2e3e525f080/

Instead, we should make servers and clients cooperate so they know this is a temporary "expected failure" (ie. similar to HTTP 503), and make clients retry for longer.

Revisions and Commits

rDSTO Storage manager
D8224rDSTO5335244fc187 retry: Add constant 10s wait when retrying transient exceptions
D8223rDSTO7c7a721da2ae Convert psycopg2 errors to TransientRemoteException instead of RemoteException
rDCORE Foundations and core functionalities
D8221rDCOREa0e61ab86973 Make the RPC client raise a specific exception class on 503
D8220rDCORE9848432cd77a Add tests for RPC server's exception handling
D8219rDCOREe6497914d42b Remove support for deprecated exception format

Event Timeline

vlorentz claimed this task.Aug 9 2022, 11:17 AM
vlorentz triaged this task as Normal priority.
vlorentz created this task.
vlorentz added revisions: D8219: Remove support for deprecated exception format, D8220: Add tests for RPC server's exception handling, D8221: Make the RPC client raise a specific exception class on 503.
vlorentz added revisions: D8223: Convert psycopg2 errors to TransientRemoteException instead of RemoteException, D8224: retry: Add constant 10s wait when retrying transient exceptions.Aug 9 2022, 4:20 PM
vlorentz added a commit: rDCOREe6497914d42b: Remove support for deprecated exception format.Aug 10 2022, 9:48 AM
vlorentz added a commit: rDCORE9848432cd77a: Add tests for RPC server's exception handling.
vlorentz added a commit: rDCOREa0e61ab86973: Make the RPC client raise a specific exception class on 503.Aug 16 2022, 1:48 PM
vlorentz added a commit: rDSTO7c7a721da2ae: Convert psycopg2 errors to TransientRemoteException instead of RemoteException.Aug 16 2022, 2:20 PM
vlorentz added a commit: rDSTO5335244fc187: retry: Add constant 10s wait when retrying transient exceptions.
vlorentz added a subtask: Restricted Maniphest Task.Aug 17 2022, 3:50 PM
vlorentz removed a subtask: Restricted Maniphest Task.Aug 24 2022, 12:35 PM
vlorentz closed this task as Resolved.Sep 15 2022, 1:56 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:37 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal