beaubourg has run out of disk space on the postgresql partition. This broke logical replication between belvedere and somerset, and probably the main indexer database too.
Ideally, now that the VM storage has been moved to ceph on dedicated SSDs, we should migrate this local postgres storage to ZFS.
One way we could do this:
- file-level copy of the data to a dedicated ceph storage space
- tear down the current lvm / raid pile
- initialize a new zfs pool
- copy the data back
We might want to also migrate the main indexer database to belvedere, as it still has some spare SSDs (T2440)