Knak experienced a critical incident on August 18, 2021, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 18, 2021, 03:59 PM UTC
Our Redis/caching clusters filled up and crashed the application. This made it such that sessions no longer worked and crashed the application altogether. Solving the crash was quick and simple, but unfortunately, our alerts didn't work as intended and it happened overnight and we didn't get to it until 4 hours later. We are investigating the root cause of the caches filling up, investigating our processes and alarms to ensure we get ahead of this type of issue happening again, and actively looking into solutions for 24-hour crisis support for incidents such as this so we can jump on them right away.