A way to mitigate the icinga apt notification spam, and also a way to reduce tedious sysadm work, is to let unattended-upgrades do its magic.
We just need to be careful on which packages are potentially dangerous for upgrades and that we do not want to be upgraded unattended — postgres comes to mind, but there might be others.
(IIRC upgrades that requires interaction, e.g., conffile changes, are blacklisted by default by unattended-upgrades, but this requires double-checking.)
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T755 icinga notification spam: pending package upgrades | ||
| Migrated | gitlab-migration | T756 switch to unattended-upgrades for (non critical) package upgrades |
(IIRC upgrades that requires interaction, e.g., conffile changes, are blacklisted by default by unattended-upgrades, but this requires double-checking.)
I already currently do 99% of package upgrades unattended using clush on louvre
My command line is :
sudo clush -b -w @all env DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y dist-upgrade
Which ignores all debconf prompts, and makes dpkg cleanly update the default configuration files while keeping modified configuration files untouched.
I checked and unattended-upgrades indeed blacklists packages whose upgrade would trigger conffile prompts, unless --force-confdef and/or --force-confold are used.
As a first pass, we should be able to whitelist the following package origins :
"o=Debian,n=${lsbdistcodename}", # main Debian archive
"o=Debian,n=${lsbdistcodename}-updates", # stable-updates (ex-volatile)
"o=Debian,n=${lsbdistcodename},l=Debian-Security", # security updates
"o=debian icinga-${lsbdistcodename},n=icinga-${lsbdistcodename}", # Icinga2 repository
"o=Proxmox,n=${lsbdistcodename}", # Proxmox repository
"o=elastic", # Yes elasticsearch has both cases <_<
"o=Elastic", # ...This would avoid autoupdates from :
which are the sources of most package updates that would need a manual interaction afterwards...
We should probably pull the following nagios checks:
https://anonscm.debian.org/cgit/mirror/dsa-nagios.git/tree/dsa-nagios-checks/checks/dsa-check-running-kernel
https://anonscm.debian.org/cgit/mirror/dsa-nagios.git/tree/dsa-nagios-checks/checks/dsa-check-libs
(yes, this will create some different nagios noise, but for stuff that probably really needs to be done by a human).
(Which we can do by setting Dpkg::Options to ["--force-confdef", "--force-confold"] in /etc/apt.conf.d/50unattended-upgrades)
That's implemented now; it's been deployed on the workers, and should kick in during the next run of the apt periodic runner (i.e. during the night).
If that succeeds i'll deploy on the rest of the hosts.