Kalix EMR Outage History

Notifications are currently not working and the queue is delayed. This means that messages will take a while to send, outgoing batches will be pending, and other fields may take a while to update. These will eventually be processed, but it will take some time until the queue is cleared. This issue is also affecting notifications, so if you want to see unread notifications, check the 'Messaging' -> 'Inbox' page directly.

If you are seeing an error when trying to load the 'Messaging' -> 'Inbox' page, try to change the filter to 'Last 6 Months' or 'Last Month' to reduce the amount of data that needs to load.

We are currently working on some database indexes to help fix up the queue (and make Kalix faster too). This process is slow though, as we identify the correct indexes to use. Please note that incoming SMS are handled in the queue, so will also be delayed.

There are issues logging in to Kalix because the authentication servers are not running. There is currently an outage at our main provider location (Azure West US), causing downtime in Kalix. We are attempting workarounds by starting new servers, but the issue appears to be communication within the datacenter. We will continue to monitor the status of the servers and take further action if it looks like this issue is going to be extended.

We have found out that the underlying issue was caused by a power failure in the datacenter. It is running on backup but there were some systems that did not recover correctly. Most of our components are live but we are still having trouble with our API which manages the backend of Kalix, this might be affected by the reduced capacity of the datacenter but we are looking to see if there is a secondary issue.

We were able to find the underlying cause of the downtime - most of our infrastructure/databases are live, but we have one service that manages our backend queues that is has been affected by the downtime. We are working with our provider to bring this component back up, or will possibly create an alternative component to use instead.

The service that is causing the current Kalix outage is experiencing downtime in the West US region of Azure due to a sudden power outage, and this issue is still being worked on. Further updates can be seen here: https://azure.status.microsoft/en-us/status While the power outage has been solved, it seems like the recovery is still underway. Attempts to bypass the component were not successful, as spinning up alternatives is also affected by the downtime in the West US region. Bypassing the region entirely will take more time as our databases would have to be replicated and recreated, so we are waiting for the West US region to recover.