Page MenuHomeSoftware Heritage

Resolve disk full issue on somerset:/srv/softwareheritage/postgres
Closed, MigratedEdits Locked

Description

somerset:/srv/softwareheritage/postgres was almost full this morning.

One of the main databases using this storage space, softwareheritage-indexer, went down.

Event Timeline

ftigeot changed the task status from Open to Work in Progress.Dec 3 2018, 3:10 PM
ftigeot triaged this task as Unbreak Now! priority.
ftigeot created this task.

somerset:softwareheritage-indexer is the master database for dbreplica1:softwareheritage-indexer.

Transactions are replicated between both hosts using the Postgres streaming replication mechanism.
somerset:softwareheritage-indexer went down due to lack of free disk space on its storage volume.
2.8TB of WAL files remained to be processed at the time.

This lack of free space was a side effect of the T1395 issue.

The WAL files are being manually copied to dbreplica1 and ingested by the Postgres instance running there.

Some no longer useful dump files were removed by seirl@, freeing some space on somerset:/srv/softwareheritage/postgres .

ftigeot claimed this task.

Service postgresql@10-indexer.service has been restarted on somerset and database replication is once again operating normally.
Postgres wal files are being removed as expected on the master, slowly freeing disk space.

gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:54 PM
gitlab-migration claimed this task.
gitlab-migration changed the status of subtask T1395: Enlarge disk on dbreplica1 from Resolved to Migrated.
gitlab-migration added a subscriber: gitlab-migration.