Page MenuHomeSoftware Heritage

Resolve disk full issue on somerset:/srv/softwareheritage/postgres
Closed, ResolvedPublic

Description

somerset:/srv/softwareheritage/postgres was almost full this morning.

One of the main databases using this storage space, softwareheritage-indexer, went down.

Event Timeline

ftigeot created this task.Dec 3 2018, 3:10 PM
ftigeot changed the task status from Open to Work in Progress.
ftigeot triaged this task as Unbreak Now! priority.

somerset:softwareheritage-indexer is the master database for dbreplica1:softwareheritage-indexer.

Transactions are replicated between both hosts using the Postgres streaming replication mechanism.
somerset:softwareheritage-indexer went down due to lack of free disk space on its storage volume.
2.8TB of WAL files remained to be processed at the time.

This lack of free space was a side effect of the T1395 issue.

The WAL files are being manually copied to dbreplica1 and ingested by the Postgres instance running there.

Some no longer useful dump files were removed by seirl@, freeing some space on somerset:/srv/softwareheritage/postgres .

ftigeot closed this task as Resolved.Dec 4 2018, 11:31 AM
ftigeot claimed this task.

Service postgresql@10-indexer.service has been restarted on somerset and database replication is once again operating normally.
Postgres wal files are being removed as expected on the master, slowly freeing disk space.