Page MenuHomeSoftware Heritage
Create Task
Maniphest T866

uffizi disk's full makes some workers fail
Closed, MigratedEdits Locked
Actions

Description

Uffizi regularly got its / disk full.
Resulting in workers (well at least the indexers) failing about OSError... no more disk space (P197 for a sample extracted from our logstash instance).

It's possibly related to T789 (quoting the first line of that issue):

We currently let the systemd journal take up as much space as it needs on the root filesystem of our machines.

And indeed, the /var/log/ (mounted on /) is quite bloated at that moment.

The workaround so far, for my part, was to:

  • claim some space in /var/cache/apt/archives
  • to permit triggering the logrotate rune to compress logs
sudo logrotate -f /etc/logrotate.conf

A better workaround (as a default for a better solution) would be to increase uffizi's disk size.

Related Objects

Event Timeline

ardumont renamed this task from uffizi disk full make workers failing to uffizi disk's full makes some workers fail.Dec 1 2017, 1:54 PM
ardumont created this task.
ardumont updated the task description. (Show Details)Dec 1 2017, 1:57 PM
ardumont updated the task description. (Show Details)Dec 1 2017, 2:01 PM
ardumont updated the task description. (Show Details)
zack raised the priority of this task from Normal to Unbreak Now!.Dec 1 2017, 2:05 PM
ftigeot closed this task as Resolved.Dec 12 2017, 4:32 PM
ftigeot claimed this task.
ftigeot added a comment.Dec 12 2017, 4:41 PM

Resolved with help from @olasd .

Steps taken:

  • Cap most log files to 100MB:
/etc/logrotate.d/rsyslog
...
+maxsize 100M
...
  • Resize base disk image using Proxmox or low-level dm commands
  • Resize / partition
apt-get install parted
parted /dev/vda
  resizepart 1
  100%
  • Resize / filesystem
resize2fs /dev/vda1
ftigeot added a comment.Dec 12 2017, 4:42 PM

Base disk image size was increased from 10 to 20GB.

olasd mentioned this in D282: db: use a thread instead of a tmpfile to COPY data to postgresql.Dec 21 2017, 7:48 PM
olasd mentioned this in rDSTOa0ac20adaa34: db: use a thread instead of a tmpfile to COPY data to postgresql.Dec 21 2017, 8:08 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:53 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: ftigeot, gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal