Kalix EMR experienced a major incident on August 22, 2022 affecting Kalix Platform and Messaging, lasting 5h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 22, 2022, 12:17 PM UTC
We are aware of "Unknown Errors" that are currently occurring on the Kalix system. It does not seem to be every request, but some requests are returning a 0 length response which is not expected and will cause the unknown error. We are currently looking into the cause of the issue - it seems to be occurring between our servers and Cloudflare, which distributes our data.
- identified Aug 22, 2022, 12:52 PM UTC
As we were suspecting, it seems like there is a problem with our provider, Cloudflare. There is currently an issue with the 'Cloudflare Tunnel' feature which we use to connect our internet network to the general internet. We use this feature to protect your data, but it seems that requests are partially making it to our server but then not back again - this means that it may seem that a piece of data has not been saved but it actually has. This issue is currently being tracked on their side as well: https://www.cloudflarestatus.com/ We are expecting a speedy resolution and will update once we have more information.
- monitoring Aug 22, 2022, 01:12 PM UTC
We have updated the connection to Cloudflare and this has seemed to fix up the issues. We are currently monitoring but please let us know if there are any ongoing issues. The status update has not been closed on their website yet, so we will continue to monitor until it is closed.
- identified Aug 22, 2022, 01:48 PM UTC
Unfortunately in our haste to fix the earlier issue we included a bug in sending emails. We are currently pushing a fix for this issue.
- monitoring Aug 22, 2022, 02:01 PM UTC
The emailing bug has also been fixed so Kalix should be back to normal now. We will be monitoring for any more issues.
- resolved Aug 22, 2022, 05:29 PM UTC
Kalix is functioning correctly and Cloudflare's notification is also closed. We are going to resolve this issue for now.