Incident description
ProxySQL keeps crashing and reloading every 5 minutes and client connections keep being dropped continuously.
Incident Severity: CRITICAL
Data Loss: YES
Timeline
| Time (UTC) | |
|---|---|
| 10 July 2021 7:13 | an upgrade from version 2.0.7 to version 2.1.1 was triggered. |
| 10 July 2018 11:00 | we started receiving notification on Telegram about CAMS not being available. |
| 10 July 2018 17:00 | Massimiliano Adamo identified an issue on ProxySQL. PRoxySQL was reload and logging was increased. |
| 12 July 2018 10:30 | Bjarke Madsen and Robert Latta sent a message to Massimiliano Adamo , to notify an issue with the Web UI of BRIAN. |
| 12 July 2018 13:30 | ProxySQL was pinned to an older version |
Total Downtime: it was an intermittent downtime of ~ 48 hours
Details of Solution
I have enabled version and version pinning in puppet to avoid future updates.
Future mitigation
In general, it's a good practice to always pin critical applications.