somerset:/srv/softwareheritage/postgres was almost full this morning.
One of the main databases using this storage space, softwareheritage-indexer, went down.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T1404 Resolve disk full issue on somerset:/srv/softwareheritage/postgres | ||
| Migrated | gitlab-migration | T1395 Enlarge disk on dbreplica1 |
Event Timeline
Comment Actions
somerset:softwareheritage-indexer is the master database for dbreplica1:softwareheritage-indexer.
Transactions are replicated between both hosts using the Postgres streaming replication mechanism.
somerset:softwareheritage-indexer went down due to lack of free disk space on its storage volume.
2.8TB of WAL files remained to be processed at the time.
This lack of free space was a side effect of the T1395 issue.
The WAL files are being manually copied to dbreplica1 and ingested by the Postgres instance running there.
Comment Actions
Some no longer useful dump files were removed by seirl@, freeing some space on somerset:/srv/softwareheritage/postgres .
Comment Actions
Service postgresql@10-indexer.service has been restarted on somerset and database replication is once again operating normally.
Postgres wal files are being removed as expected on the master, slowly freeing disk space.