Uffizi regularly got its / disk full.
Resulting in workers (well at least the indexers) failing about OSError... no more disk space (P197 for a sample extracted from our logstash instance).
It's possibly related to T789 (quoting the first line of that issue):
We currently let the systemd journal take up as much space as it needs on the root filesystem of our machines.
And indeed, the /var/log/ (mounted on /) is quite bloated at that moment.
The workaround so far, for my part, was to:
- claim some space in /var/cache/apt/archives
- to permit triggering the logrotate rune to compress logs
sudo logrotate -f /etc/logrotate.conf
A better workaround (as a default for a better solution) would be to increase uffizi's disk size.