Page MenuHomeSoftware Heritage

Migrate dbreplica0.euwest.azure to zfs
Started, Work in Progress, HighPublic

Description

dbreplica0.euwest.azure ran out of disk space.

somerset is piling up WALs waiting on this machine to get space to apply them.

Instead of adding more disks to the current lvm, we should move the data to zfs.

Event Timeline

olasd changed the task status from Open to Work in Progress.Tue, Sep 8, 3:16 PM
olasd triaged this task as High priority.
olasd created this task.
olasd added a comment.Tue, Sep 8, 3:26 PM
  • enabled non-free
  • installed build-essential
  • installed linux-image-cloud-amd64, linux-headers-cloud-amd64 from bpo
  • installed zfs-dkms from bpo
  • use resource disk as cache:
    • in /etc/waagent.conf, set Resourcedisk.Format=n
    • prepare /etc/systemd/system/zfs-cache-azure.service
[Unit]
After=zfs-import.target
Before=zfs-mount.target
Wants=zfs-import.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStartPre=-zpool remove data sdb
ExecStart=zpool add data cache sdb
ExecStop=zpool remove data sdb

[Install]
WantedBy=multi-user.target
  • reboot on bpo kernel with zfs module enabled
olasd added a comment.EditedTue, Sep 8, 4:34 PM

I've replaced zfs-cache-azure.service with the following:

[Unit]
After=zfs-import.target
Before=zfs-mount.target
Wants=zfs-import.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStartPre=-zpool remove data /dev/disk/azure/resource
ExecStart=zpool add data cache /dev/disk/azure/resource
ExecStop=zpool remove data /dev/disk/azure/resource

[Install]
WantedBy=multi-user.target

(turns out the resource disk pops up as sda on this machine...)

For now, the machine is set up with the following layout:

  • 2 special allocation class devices, 256GiB each
  • 6 x 1 TB HDDs for data
  • the azure resource disk as L2ARC

I've followed the instructions on https://www.percona.com/live/17/sites/default/files/slides/PostgreSQL%20%2B%20ZFS%20Best%20Practices.pdf for the dataset tuning - slide 82 onwards

.

This layout manages 300 MB/s write speeds, we'll see if that's enough...

olasd added a comment.Wed, Sep 9, 10:35 AM

The migration of the data to zfs is complete, and the database has been restarted.

The old LVM now needs to be cleaned up.