We use let's encrypt certs for the kafka clusters. These certs have a short expiry period, so we need to update them regularly.
The automatic updates happen (30 days before expiry) but the kafka service must be notified that the update happened. Apparently, it currently doesn't.
kafka can reload its keystore dynamically.
For each listener with TLS enabled, you need to call the following command:
/opt/kafka/bin/kafka-configs.sh --bootstrap-server {broker_address} --entity-name {broker_id} --entity-type brokers --add-config listener.name.{listener_name}.ssl.keystore.location={keystore_location} --alterThis makes the listener reload the key from the keystore.
Since we've had another certificate expiry happen in the prod kafkas, I've finally deployed D5864 (and a followup syntax fix, no thanks to octocatalog-diff).
After removing the keystores on all kafka nodes, a puppet run has restored them and the certificates have been reloaded properly (without disruption to kafka clients, afaict).
After testing that, I've also forced the renewal of one of the certificates (on kafka1). The next puppet run properly updated the certificate, and made the broker load the new certificate.
I'll try to remember to force the renewal on the other nodes in a few days, so that we avoid having all nodes update their certs at the same time every two months.