This seems to have happened after this weeks' reboots, probably the DNAT didn't come back properly.
Note that this gets in the way of the weekly postgres base backup, which indeed didn't happen tonight:
2015-12-12 00:31:01,366  barman.server ERROR: Check 'ssh' failed for server 'swh'
2015-12-12 00:31:01,394  barman.server ERROR: Impossible to start the backup. Check the log for more details, or run 'barman check swh'
"barman check swh" is enough to verify that the ssh works