Page MenuHomeSoftware Heritage

Out of disk space on prometheus storage
Closed, MigratedEdits Locked

Description

The partition used to store the prometheus storage is almost full

vsellier@pergamon /var/lib/prometheus/metrics2
 % df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc1       246G  228G  5.9G  98% /var/lib/prometheus

The disk should be extended to avoid a disruption of service

Event Timeline

vsellier created this task.
vsellier changed the task status from Open to Work in Progress.Dec 7 2020, 12:38 PM
vsellier claimed this task.

The disk will be resized to avoid the service disruption in a short term before looking at T1362

  • an increment of 250Go was added via the proxmox ui (for a total of 500Go now)
  • the disk was resized on the os side :
root@pergamon:~# parted /dev/vdc
GNU Parted 3.2
Using /dev/vdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 537GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  268GB  268GB  ext4

(parted) print free                                                       
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 537GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  268GB   268GB   ext4
        268GB   537GB   268GB   Free Space

(parted) resizepart 1                                                     
Warning: Partition /dev/vdc1 is being used. Are you sure you want to continue?
Yes/No? yes                                                               
End?  [268GB]? 537GB                                                      
(parted) print                                                            
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 537GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  537GB  537GB  ext4

(parted) print free                                                       
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 537GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  537GB   537GB   ext4


##### FS resize
root@pergamon:~# resize2fs /dev/vdc1
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/vdc1 is mounted on /var/lib/prometheus; on-line resizing required
old_desc_blocks = 32, new_desc_blocks = 63
The filesystem on /dev/vdc1 is now 131071739 (4k) blocks long.

root@pergamon:~# df -h /var/lib/prometheus
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc1       492G  228G  242G  49% /var/lib/prometheus