Following [1], the riverside node now uses its own kafka.
Its disk grew a bit fast, so we need to extend it a bit [2]
[1] T2888
[2]
root@riverside:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/riverside--vg-root 28G 26G 576M 98% /
Extended the disk / to 32Gib more in the proxmox ui.
Then ssh into the node and extend the partition;
root@riverside:~# pvscan
PV /dev/sda1 VG riverside-vg lvm2 [<32.00 GiB / 4.00 MiB free]
Total: 1 [<32.00 GiB] / in use: 1 [<32.00 GiB] / in no VG: 0 [0 ]
root@riverside:~# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 34.4GB 34.4GB primary boot, lvm
(parted) print free
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
32.3kB 1049kB 1016kB Free Space
1 1049kB 34.4GB 34.4GB primary boot, lvm
34.4GB 68.7GB 34.4GB Free Space
(parted) resizepart 1
End? [34.4GB]? 68.7GB
(parted) p
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 68.7GB 68.7GB primary boot, lvm
root@riverside:~# pvresize /dev/sda1 --test
TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
Physical volume "/dev/sda1" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
root@riverside:~# pvscan
PV /dev/sda1 VG riverside-vg lvm2 [<63.98 GiB / 31.98 GiB free]
Total: 1 [<63.98 GiB] / in use: 1 [<63.98 GiB] / in no VG: 0 [0 ]
root@riverside:~# lvresize --extents +100%FREE --resizefs /dev/mapper/riverside--vg-root
Size of logical volume riverside-vg/root changed from <28.18 GiB (7214 extents) to 60.16 GiB (15402 extents).
Logical volume riverside-vg/root successfully resized.
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mapper/riverside--vg-root is mounted on /; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 8
The filesystem on /dev/mapper/riverside--vg-root is now 15771648 (4k) blocks long.And then eventually starting back what went down:
root@riverside:/var/lib/sentry-on-premise# docker-compose up -d
And somehow, we needed to do so sentry kept up:
root@riverside:/var/lib/sentry-onpremise# docker-compose up -d --force-recreate --no-deps redis
Monitoring a bit everything is getting back to normal
(we don't see new events in the interface right now, might be taking some time to consume back messages from the kafka topics).
Unrelated to this task, sentry seems no longer consuming, that's T2899.
So closing this now.