Page MenuHomeSoftware Heritage

Add throttling/backoff to origin visit scheduler respawn logic
Open, NormalPublic

Description

When respawning the scheduling threads, no throttling is applied. This makes the process respawn threads (very) fast when the DB is down.

https://sentry.softwareheritage.org/organizations/swh/issues/10223/?referrer=phabricator_plugin

OperationalError: connection to server at "db1.internal.staging.swh.network" (192.168.130.11), port 5432 failed: FATAL:  server login has been failing, try again later (server_login_retry)
connection to server at "db1.internal.staging.swh.network" (192.168.130.11), port 5432 failed: FATAL:  server login has been failing, try again later (server_login_retry)

(2 additional frame(s) were not displayed)
...
  File "swh/scheduler/backend.py", line 72, in __init__
    cursor_factory=psycopg2.extras.RealDictCursor,
  File "psycopg2/pool.py", line 162, in __init__
    self, minconn, maxconn, *args, **kwargs)
  File "psycopg2/pool.py", line 59, in __init__
    self._connect()
  File "psycopg2/pool.py", line 63, in _connect
    conn = psycopg2.connect(*self._args, **self._kwargs)
  File "__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

Thread directory died with exception; respawning

Event Timeline

olasd triaged this task as Normal priority.
olasd added a project: Easy hack.