Incident Description
Users of EMS could not logon to EMS. They were presented with the login screen, which took them to the IDP selection page (as per normal). After successful authentication on the IDP, they where redirected to EMS. However, instead of being logged in on EMS, they where logged out.
The reason for degradation:
- EMS/Indico stores user sessions in redis
- prod-events01.geant.org and prod-events02.geant.org could not resolve the hostname for the redis server (master.production-events-redis.service.ha.geant.net)
- With the connection to redis lost, Indico could not create or manage user sessions
The impact of this service degradation was:
- Users could not manage their events, for example:
- Editing the event
- Accessing registation lists
- Sending out reminder emails
Incident severity: MED Partial service degradation
Data loss: NO
Total duration of incident: 15 hours
Timeline
All times are in UTC
| Date | Time | Description |
|---|---|---|
| 21:55:53 | First error in indico.log of redis being unavailable: ConnectionError: Error -2 connecting to master.production-events-redis.service.ha.geant.net:6379. Name or service not known. |
| 10:42 | First user query about EMS login problem (Slack #general) |
| 11:14 | Ian Galpin identified the dns resolution problem [root@prod-events01 log]# ping master.production-events-redis.service.ha.geant.net ping: master.production-events-redis.service.ha.geant.net: Name or service not known |
| 12:06 | Service degradation incident email sent out to product owner (Steffie Bosman) |
| 12:12 | Massimiliano Adamo identified a problem with powerDNS [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net slave.production-events-redis.service.ha.geant.net has address 83.97.94.19 [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net slave.production-events-redis.service.ha.geant.net has address 83.97.94.19 Host slave.production-events-redis.service.ha.geant.net not found: 3(NXDOMAIN) [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net Host slave.production-events-redis.service.ha.geant.net not found: 3(NXDOMAIN) [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net Host slave.production-events-redis.service.ha.geant.net not found: 3(NXDOMAIN) [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net Host slave.production-events-redis.service.ha.geant.net not found: 3(NXDOMAIN) [root@prod-events02 ~]# host slave.production-events-redis.service.ha.geant.net slave.production-events-redis.service.ha.geant.net has address 83.97.94.19 consul DNS resolution seemed to work: dig slave.production-events-redis.service.ha.geant.net @prod-consul01.geant.org -p 8600 dig slave.production-events-redis.service.ha.geant.net @prod-consul02.geant.org -p 8600 dig slave.production-events-redis.service.ha.geant.net @prod-consul03.geant.org -p 8600 |
| 12:30 | Massimiliano Adamo resolved the PowerDNS issue by disabling the packetcache config option:
The following GitHub issue might explain the issue: https://github.com/PowerDNS/pdns/issues/8160 |
| 13:01 | Service restored email sent out to product owner (Steffie Bosman) |
Proposed Solution
- Additional monitoring (Sensu checks) will be added
- These will check that specific hostnames resolve
- This is an action item for DevOps team
- the issue was solved by disabling Packet Cache on PowerDNS, which is enabled by default: https://docs.powerdns.com/recursor/settings.html#disable-packetcache