Kalix EMR experienced a minor incident on January 29, 2024 affecting Kalix Platform and Online Schedulers, lasting 5h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 29, 2024, 07:44 PM UTC
There is an increased incidence of time-outs on Kalix's database, resulting in pages taking longer to load and erroring when loading. Creating documents is especially affected due to its higher load. We have contacted our cloud storage provider for assistance.
- identified Jan 29, 2024, 07:44 PM UTC
We are continuing to work on a fix for this issue.
- identified Jan 29, 2024, 08:59 PM UTC
We have identified an indexing issue with the reminder messages, which is having a flow-on effect on the rest of the database. We are currently pushing up a fix to include an index that should significantly help with the performance of these queries, which in turn should improve the rest of Kalix. It may take a few hours for this index to propagate fully, but we are monitoring the queries and will provide an update as soon as we think this has made a difference or not.
- resolved Jan 30, 2024, 01:37 AM UTC
The indexing fix completed approximately 3 hours ago, and this seems to have resolved the timeout issues from that point onwards. We will be resolving this issue for now.