Change Details

UTC+2 time -- Sun, 25 Jul 2021 -- 03:27 <+swhbot> icinga PROBLEM: service linux-ssh on worker03.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:30 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is WARNING: WARNING - load average: 36.25, 23.93, 12.73 03:30 <+swhbot> icinga PROBLEM: service journalbeat on worker03.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:36 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is CRITICAL: CRITICAL - load average: 50.69, 40.68, 23.70 03:40 <+swhbot> icinga PROBLEM: service mediawiki wiki.softwareheritage.org on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh hedgedoc on rp1.internal.admin.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service mediawiki intranet.softwareheritage.org + auth on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp content_known on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision log end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp counters on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service phabricator on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp snapshot end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh-storage api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp directory end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp release end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh-search api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp content end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service opnsense on pushkin.internal.softwareheritage.org is CRITICAL: connect to address pushkin.internal.softwareheritage.org and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket 03:42 <+swhbot> icinga PROBLEM: service ping4 on pushkin.internal.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.2) 03:42 <+swhbot> icinga PROBLEM: service linux-ssh on worker06.softwareheritage.org is CRITICAL: connect to address 192.168.100.26 and port 22: No route to host 03:42 <+swhbot> icinga PROBLEM: service ping4 on worker06.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.26) 03:42 <+swhbot> icinga PROBLEM: host pushkin.internal.softwareheritage.org is DOWN: CRITICAL - Host Unreachable (192.168.100.2) 03:43 <+swhbot> icinga PROBLEM: service puppet_agent on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service load on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service linux-ssh on worker08.softwareheritage.org is CRITICAL: connect to address 192.168.100.28 and port 22: No route to host 03:43 <+swhbot> icinga PROBLEM: service disk / on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on pergamon.softwareheritage.org is WARNING: WARNING - load average: 33.75, 22.42, 10.55 03:44 <+swhbot> icinga PROBLEM: service journalbeat on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service apt on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service load on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ping4 on worker08.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.28) 03:45 <+swhbot> icinga PROBLEM: service disk / on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk /boot on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker02.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:45 <+swhbot> icinga PROBLEM: service ping4 on softwareheritage.org is UNKNOWN: check_ping: Invalid hostname/address - softwareheritage.orgUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 03:45 <+swhbot> icinga PROBLEM: service journalbeat on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk / on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ntp on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:46 <+swhbot> icinga PROBLEM: service journalbeat on worker15.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service journalbeat on tate.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service sentry https certificate on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service icingaweb2 http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service swh-storage api (localhost) on webapp.internal.staging.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service grafana https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service sentry http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service bitbucket_archive https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds

UTC+2 time -- Mon, 26 Jul 2021 -- 03:27 <+swhbot> icinga PROBLEM: service linux-ssh on worker03.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:30 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is WARNING: WARNING - load average: 36.25, 23.93, 12.73 03:30 <+swhbot> icinga PROBLEM: service journalbeat on worker03.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:36 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is CRITICAL: CRITICAL - load average: 50.69, 40.68, 23.70 03:40 <+swhbot> icinga PROBLEM: service mediawiki wiki.softwareheritage.org on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh hedgedoc on rp1.internal.admin.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service mediawiki intranet.softwareheritage.org + auth on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp content_known on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision log end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp counters on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service phabricator on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp snapshot end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh-storage api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp directory end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp release end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh-search api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp content end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service opnsense on pushkin.internal.softwareheritage.org is CRITICAL: connect to address pushkin.internal.softwareheritage.org and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket 03:42 <+swhbot> icinga PROBLEM: service ping4 on pushkin.internal.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.2) 03:42 <+swhbot> icinga PROBLEM: service linux-ssh on worker06.softwareheritage.org is CRITICAL: connect to address 192.168.100.26 and port 22: No route to host 03:42 <+swhbot> icinga PROBLEM: service ping4 on worker06.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.26) 03:42 <+swhbot> icinga PROBLEM: host pushkin.internal.softwareheritage.org is DOWN: CRITICAL - Host Unreachable (192.168.100.2) 03:43 <+swhbot> icinga PROBLEM: service puppet_agent on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service load on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service linux-ssh on worker08.softwareheritage.org is CRITICAL: connect to address 192.168.100.28 and port 22: No route to host 03:43 <+swhbot> icinga PROBLEM: service disk / on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on pergamon.softwareheritage.org is WARNING: WARNING - load average: 33.75, 22.42, 10.55 03:44 <+swhbot> icinga PROBLEM: service journalbeat on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service apt on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service load on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ping4 on worker08.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.28) 03:45 <+swhbot> icinga PROBLEM: service disk / on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk /boot on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker02.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:45 <+swhbot> icinga PROBLEM: service ping4 on softwareheritage.org is UNKNOWN: check_ping: Invalid hostname/address - softwareheritage.orgUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 03:45 <+swhbot> icinga PROBLEM: service journalbeat on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk / on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ntp on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:46 <+swhbot> icinga PROBLEM: service journalbeat on worker15.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service journalbeat on tate.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service sentry https certificate on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service icingaweb2 http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service swh-storage api (localhost) on webapp.internal.staging.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service grafana https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service sentry http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service bitbucket_archive https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds

UTC+2 time -- Sun-- Mon, 2526 Jul 2021 -- 03:27 <+swhbot> icinga PROBLEM: service linux-ssh on worker03.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:30 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is WARNING: WARNING - load average: 36.25, 23.93, 12.73 03:30 <+swhbot> icinga PROBLEM: service journalbeat on worker03.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:36 <+swhbot> icinga PROBLEM: service load on worker03.softwareheritage.org is CRITICAL: CRITICAL - load average: 50.69, 40.68, 23.70 03:40 <+swhbot> icinga PROBLEM: service mediawiki wiki.softwareheritage.org on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh hedgedoc on rp1.internal.admin.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service mediawiki intranet.softwareheritage.org + auth on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp content_known on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision log end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp revision end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp counters on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service phabricator on tate.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp snapshot end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh-storage api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh deposit on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:41 <+swhbot> icinga PROBLEM: service swh webapp directory end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp release end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh-search api (remote) on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp content end to end on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service swh webapp http redirect on moma.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:42 <+swhbot> icinga PROBLEM: service opnsense on pushkin.internal.softwareheritage.org is CRITICAL: connect to address pushkin.internal.softwareheritage.org and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket 03:42 <+swhbot> icinga PROBLEM: service ping4 on pushkin.internal.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.2) 03:42 <+swhbot> icinga PROBLEM: service linux-ssh on worker06.softwareheritage.org is CRITICAL: connect to address 192.168.100.26 and port 22: No route to host 03:42 <+swhbot> icinga PROBLEM: service ping4 on worker06.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.26) 03:42 <+swhbot> icinga PROBLEM: host pushkin.internal.softwareheritage.org is DOWN: CRITICAL - Host Unreachable (192.168.100.2) 03:43 <+swhbot> icinga PROBLEM: service puppet_agent on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service load on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:43 <+swhbot> icinga PROBLEM: service linux-ssh on worker08.softwareheritage.org is CRITICAL: connect to address 192.168.100.28 and port 22: No route to host 03:43 <+swhbot> icinga PROBLEM: service disk / on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on pergamon.softwareheritage.org is WARNING: WARNING - load average: 33.75, 22.42, 10.55 03:44 <+swhbot> icinga PROBLEM: service journalbeat on worker06.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker06.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service apt on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service ntp on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:44 <+swhbot> icinga PROBLEM: service load on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service load on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ping4 on worker08.softwareheritage.org is CRITICAL: CRITICAL - Host Unreachable (192.168.100.28) 03:45 <+swhbot> icinga PROBLEM: service disk / on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk /boot on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker08.softwareheritage.org is UNKNOWN: Remote Icinga instance 'worker08.softwareheritage.org' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service journalbeat on worker02.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:45 <+swhbot> icinga PROBLEM: service ping4 on softwareheritage.org is UNKNOWN: check_ping: Invalid hostname/address - softwareheritage.orgUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4 03:45 <+swhbot> icinga PROBLEM: service journalbeat on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service disk / on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service ntp on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:45 <+swhbot> icinga PROBLEM: service puppet_agent on rp1.internal.admin.swh.network is UNKNOWN: Remote Icinga instance 'rp1.internal.admin.swh.network' is not connected to 'pergamon.softwareheritage.org' 03:46 <+swhbot> icinga PROBLEM: service journalbeat on worker15.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service journalbeat on tate.softwareheritage.org is UNKNOWN: <Timeout exceeded.> 03:47 <+swhbot> icinga PROBLEM: service sentry https certificate on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service icingaweb2 http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service swh-storage api (localhost) on webapp.internal.staging.swh.network is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service grafana https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service sentry http redirect on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds 03:47 <+swhbot> icinga PROBLEM: service bitbucket_archive https on pergamon.softwareheritage.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds