...
Incident severity: CRITICAL
Data loss: NO
Timeline
| Time (CET) | |
|---|---|
| 12:10 | Apache server stop accepting incoming requests |
| 12:12 | Chris Atherton reported on #it channel that site aac-project.eu is not working correnctly |
| 12:21 | Konstantin Lepikhov confirmed the issue with worldpress1 site on #devops channel |
| 12:23 | Dick Visser connected to VM via console and confirmed that network is down (router not reachable) |
| 12:29 | Massimiliano Adamo have restarted network service inside VM, after that everything started working and network link came up. |
| 12:30 | Konstantin Lepikhov announced that problem fixed. |
Total downtime: 20 minutes.
...
We're currently investigating the nature of the issue. I could either VMWware network adapter issue or something related network configuration in VMWhare cluster.
Excerpts from apache logs:
wordpress1.geant.org:80 2018-02-22 12:09:47.449513 ::1 "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
wordpress1.geant.org:80 2018-02-22 12:09:48.450712 ::1 "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
wordpress1.geant.org:80 2018-02-22 12:09:49.451963 ::1 "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
wordpress1.geant.org:80 2018-02-22 12:09:50.453232 ::1 "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
wordpress1.geant.org:80 2018-02-22 12:09:51.454511 ::1 "OPTIONS * HTTP/1.0" 200 110 "-" "Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g (internal dummy connection)"
wordpress1.geant.org:80 2018-02-22 12:09:51.887682 105.224.240.200 "GET / HTTP/1.1" 302 441 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
....
wordpress1.geant.org:80 2018-02-22 12:29:22.163447 158.227.123.38 "GET /?p=europe HTTP/1.1" 302 459 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
www.eduroam.org:443 2018-02-22 12:29:22.132767 92.40.249.120 "GET /about/connect-yourself/ HTTP/1.1" 200 13003 "https://www.google.co.uk/" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/64.0.3282.112 Mobile/14D27 Safari/602.1"
www.eduroam.org:443 2018-02-22 12:29:22.418523 92.40.249.120 "GET /wp-includes/js/wp-emoji-release.min.js?ver=4.9.4 HTTP/1.1" 200 4662 "https://www.eduroam.org/about/connect-yourself/" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/64.0.3282.112 Mobile/14D27 Safari/602.1"
www.eduroam.org:443 2018-02-22 12:29:22.493989 92.40.249.120 "GET /wp-content/plugins/templatesnext-toolkit/css/animate.css?ver=3.5.1 HTTP/1.1" 200 4905 "https://www.eduroam.org/about/connect-yourself/" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/64.0.3282.112 Mobile/14D27 Safari/602.1"
Monitoring alerted: YES