Kalix EMR Outage History

Started Jun 27, 2026, 03:15 PM UTC· resolved Jun 28, 2026, 05:50 PM UTC

1. investigating Jun 27, 2026, 03:15 PM UTC

   We are currently investigating an issue where our database is resetting continuously, which is causing timeouts to occur on Kalix. Right now, the timeouts seem are happening in waves and a refresh should recover once the database is back. We are currently investigating why this issue is occurring, as we cannot see an obvious spike in usage that would overload the database.

2. investigating Jun 27, 2026, 04:23 PM UTC

   We have tried to scale up the database; however, this process is currently stuck. It seems that there is a platform issue, so we have escalated this with the highest priority.

3. investigating Jun 27, 2026, 05:14 PM UTC

   The issue has been confirmed by Microsoft and has been escalated to an engineer. We will be continuing to work with them to get the database issue resolved.

4. investigating Jun 27, 2026, 06:18 PM UTC

   We have confirmation that Microsoft engineers are actively working on the database, and according to their latest update, they are currently investigating a specific issue. We will make further updates once we get more information.

5. monitoring Jun 27, 2026, 08:40 PM UTC

   A fix was implemented and the database is now stable again. We are currently monitoring to ensure it all looks OK.

6. resolved Jun 28, 2026, 05:50 PM UTC

   We haven't seen any ongoing issues since the database was fixed, so will close this incident.

Read the full incident report →

Started Mar 23, 2026, 06:47 PM UTC· resolved Mar 24, 2026, 05:00 AM UTC

1. identified Mar 23, 2026, 06:47 PM UTC

   Notifications are currently not working and the queue is delayed. This means that messages will take a while to send, outgoing batches will be pending, and other fields may take a while to update. These will eventually be processed, but it will take some time until the queue is cleared. This issue is also affecting notifications, so if you want to see unread notifications, check the 'Messaging' -> 'Inbox' page directly.

2. identified Mar 23, 2026, 06:59 PM UTC

   If you are seeing an error when trying to load the 'Messaging' -> 'Inbox' page, try to change the filter to 'Last 6 Months' or 'Last Month' to reduce the amount of data that needs to load.

3. identified Mar 23, 2026, 08:44 PM UTC

   We are currently working on some database indexes to help fix up the queue (and make Kalix faster too). This process is slow though, as we identify the correct indexes to use. Please note that incoming SMS are handled in the queue, so will also be delayed.

4. monitoring Mar 23, 2026, 10:12 PM UTC

   The queue is currently draining, so functionality should start to restore. Previously queued items will be processed first, and the delay of new tasks will start to shorten.

5. monitoring Mar 24, 2026, 12:09 AM UTC

   The queue has fully drained, so most of the functionality should be as normal in Kalix. We are still analyzing the usage and making improvements to the performance of the site.

6. resolved Mar 24, 2026, 05:00 AM UTC

   We will continue to monitor the platform to ensure that the queue will not get saturated tomorrow.

Read the full incident report →

Started Feb 07, 2026, 09:38 AM UTC· resolved Feb 09, 2026, 11:40 PM UTC

1. identified Feb 07, 2026, 09:38 AM UTC

   There are issues logging in to Kalix because the authentication servers are not running. There is currently an outage at our main provider location (Azure West US), causing downtime in Kalix. We are attempting workarounds by starting new servers, but the issue appears to be communication within the datacenter. We will continue to monitor the status of the servers and take further action if it looks like this issue is going to be extended.

2. identified Feb 07, 2026, 12:28 PM UTC

   We have found out that the underlying issue was caused by a power failure in the datacenter. It is running on backup but there were some systems that did not recover correctly. Most of our components are live but we are still having trouble with our API which manages the backend of Kalix, this might be affected by the reduced capacity of the datacenter but we are looking to see if there is a secondary issue.

3. investigating Feb 07, 2026, 01:36 PM UTC

   We have managed to narrow down the issue to the connection between the API server and the database, but we are having trouble working out why these two components are not able to talk to each other. Our current theory is that the power outage affected the network. We are continuing to investigate.

4. investigating Feb 07, 2026, 02:47 PM UTC

   After more investigation we have found that this is not a database issue, but rather that the application is not properly starting up (but after the database connection step). We are currently pushing different versions to try and work out the spot that is blocking the start-up.

5. identified Feb 07, 2026, 04:18 PM UTC

   We were able to find the underlying cause of the downtime - most of our infrastructure/databases are live, but we have one service that manages our backend queues that is has been affected by the downtime. We are working with our provider to bring this component back up, or will possibly create an alternative component to use instead.

6. identified Feb 07, 2026, 05:06 PM UTC

   The service that is causing the current Kalix outage is experiencing downtime in the West US region of Azure due to a sudden power outage, and this issue is still being worked on. Further updates can be seen here: https://azure.status.microsoft/en-us/status While the power outage has been solved, it seems like the recovery is still underway. Attempts to bypass the component were not successful, as spinning up alternatives is also affected by the downtime in the West US region. Bypassing the region entirely will take more time as our databases would have to be replicated and recreated, so we are waiting for the West US region to recover.

7. monitoring Feb 07, 2026, 06:11 PM UTC

   The affected component has just been restored, so Kalix is operational again. We are monitoring to make sure that all systems are functioning as expected.

8. resolved Feb 09, 2026, 11:40 PM UTC

   There has been no ongoing issues with Kalix since the component recovered.

Read the full incident report →

Started Nov 18, 2025, 11:55 AM UTC· resolved Nov 18, 2025, 09:43 PM UTC

1. identified Nov 18, 2025, 11:55 AM UTC

   There is a major Cloudflare outage which handles traffic for a large number of webpages on the internet, and this has also affected Kalix, as we use Cloudflare as our internet provider. This will probably make the news. Cloudflare's status page has been updated to show the downtime. https://www.cloudflarestatus.com/ Now that Cloudflare is looking into the issue, we are not expecting a long downtime, but we are monitoring the situation to see if an alternative option is needed.

2. monitoring Nov 18, 2025, 12:20 PM UTC

   It appears that Kalix is being reinstated in several locations - it seems like a fix is being rolled out. We will resolve this issue once we have seen an update on Cloudflare's dashboard.

3. monitoring Nov 18, 2025, 01:44 PM UTC

   Services seem to be mostly restored, without ongoing issues, but we are waiting for a confirmation from the cloudflare dashboard.

4. monitoring Nov 18, 2025, 02:47 PM UTC

   Cloudflare has given an update that the issue should be resolved in all cases, but the issue is still open for monitoring. We will also continue to monitor for any further issues.

5. resolved Nov 18, 2025, 09:43 PM UTC

   There does not seem to be ongoing issues. Cloudflare have closed the issue but have not provided a post-mortem at this time. We will be looking out for this.

Read the full incident report →