dbreplica1 overflowed its storage. We need to add some more
Description
Description
Status | Assigned | Task | ||
---|---|---|---|---|
Migrated | gitlab-migration | T1404 Resolve disk full issue on somerset:/srv/softwareheritage/postgres | ||
Migrated | gitlab-migration | T1395 Enlarge disk on dbreplica1 |
Event Timeline
Comment Actions
Target size: 8 TB striped across 8 disks (from 2 TB on a single disk).
First off the machine was too small. Resized from E2s_v2 (4 data disks) to E8s_v3 (16 data disks).
Disk creation:
for i in {1..8}; do az vm disk attach -g euwest-db --vm-name dbreplica1 --disk dbreplica1_pgdata$i --new --size-gb 1024; done
[wait for disks to appear; /dev/sdd to /dev/sdk appear]
Create a template with fdisk (single partition, full disk, type=linux raid autodetect (fd)) then duplicate it
fdisk /dev/sdd for disk in {e..k}; do sfdisk -d /dev/sdd | sfdisk /dev/sd$disk; done
Create a raid between the disks
mdadm --create /dev/md0 --level 0 --raid-devices 8 /dev/sd{d..k}1
Add the raid to the LVM volume
pvcreate /dev/md0 vgextend data-vg01 /dev/md0
Move data from the old PV to the new PV
pvmove /dev/sdc /dev/md0
[in progress...]
Comment Actions
The pvmove command was done this morning.
Next steps I did pairing with ardumont@:
- Remove old physical device vgreduce -a data-vg01
- Extend logical volume group lvextend -l"+100%pvs" /dev/mapper/data--vg01-data--lv01 /dev/md0
- Resize the filesystem resize2fs /dev/mapper/data--vg01-data--lv01