Kalix EMR experienced a critical incident on February 12, 2024 affecting Kalix Platform and Online Schedulers and 1 more component, lasting 6h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 12, 2024, 07:16 PM UTC
Kalix is experiencing loading timeout errors. We are currently investigating the cause and will have an update ASAP
- monitoring Feb 12, 2024, 07:28 PM UTC
Kalix is operational again.
- monitoring Feb 12, 2024, 07:29 PM UTC
We are continuing to monitor for any further issues.
- monitoring Feb 12, 2024, 07:53 PM UTC
This is a quick write-up of the underlying issue. The problem was with our secondary storage system (specifically, the location that stores audit history etc) that became unresponsive. We are currently working with our provider to find out more information. This is not the same as the database timeouts in our previous issue, and so far it seems to be a one-off issue in the underlying storage system as opposed to any systematic issue in Kalix, which is why it was able to recover quickly. We will continue to monitor Kalix to see if there are any ongoing issues.
- resolved Feb 13, 2024, 01:46 AM UTC
There have been no further problems with the storage, so we are closing this issue for now.