Kalix EMR experienced a major incident on April 26, 2022 affecting Kalix Platform and Messaging, lasting 10h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 26, 2022, 12:31 PM UTC
There is currently an issue loading documents (including templates) and reminders. Some of these queries may work, but there may also be timeouts. Certain queries are more affected than others, depending on how they are loaded, and it seems like templates and reminders are particularly affected. The underlying cause is that an indexing maintenance job was unsuccessful and now the index for these records has to be rebuilt. Until these indexes are complete, the queries loading the data will be inefficient and may take too long to complete and will come up with a timeout error. Unfortunately, it may take an hour or two for these indexing tasks to be completed. We will update this incident once we have more idea of the progress.
- identified Apr 26, 2022, 01:03 PM UTC
There are 4 indexes that need to be completed - docs, appointments, bills and reminders. Both appointments and reminders are over 50% complete. We will update this status when we have more progress.
- identified Apr 26, 2022, 01:57 PM UTC
The appointments and reminders indexing have completed! Out of the two remaining indexes, bills are 80% complete and documents are 30% complete. We expect once the indexing steps are complete that the timeouts should stop. We will update this status again once we have more progress.
- monitoring Apr 26, 2022, 03:28 PM UTC
From our own testing, it seems the server is starting to become more responsive as the indexing is coming to a close. The only index left is to do with documents. Document templates seem to be loading snappily in our test account.
- monitoring Apr 26, 2022, 09:36 PM UTC
All Indexing is complete and we are currently confirming performance across the platform.
- resolved Apr 26, 2022, 10:38 PM UTC
Kalix seems to be functioning normally