This seems to have happened after this weeks' reboots, probably the DNAT didn't come back properly.
Note that this gets in the way of the weekly postgres base backup, which indeed didn't happen tonight:
2015-12-12 00:31:01,366 [94490] barman.server ERROR: Check 'ssh' failed for server 'swh' 2015-12-12 00:31:01,394 [94490] barman.server ERROR: Impossible to start the backup. Check the log for more details, or run 'barman check swh'
"barman check swh" is enough to verify that the ssh works