Kalix EMR incident

Increase in timeouts in database

Minor Resolved View vendor source →

Kalix EMR experienced a minor incident on June 19, 2022 affecting Kalix Platform, lasting 3d 11h. The incident has been resolved; the full update timeline is below.

Started
Jun 19, 2022, 05:27 PM UTC
Resolved
Jun 23, 2022, 05:14 AM UTC
Duration
3d 11h
Detected by Pingoru
Jun 19, 2022, 05:27 PM UTC

Affected components

Kalix Platform

Update timeline

  1. investigating Jun 19, 2022, 05:27 PM UTC

    We have picked up an increase in errors coming from our database, that corresponded to an access outage with our provider. It looks like we were not affected by the original issue, but it seems that we have been getting increased timeouts ever since the original issue occurred. We are currently investigating the issue with our provider and hope to get the database back to a healthy state soon. The problem seems to be only with a fraction of requests to Kalix, so it doesn't seem to be a complete outage, however random actions in Kalix may have an error as this issue continues. We will update this page once we have had an update.

  2. investigating Jun 20, 2022, 11:54 AM UTC

    We are continuing to investigate this issue, we are having problems finding the underlying cause since the weekend started and are still working with our provider. We will update this page once we have more information.

  3. monitoring Jun 21, 2022, 01:41 AM UTC

    The database has been running smoothly for the last few hours without any incident, it looks like the underlying issue has been resolved but we are waiting for a confirmation from our provider. We will continue to monitor in case something changes. For the moment it seems Kalix is fully operational.

  4. resolved Jun 23, 2022, 05:14 AM UTC

    This issue seems to be resolved as we have not seen any follow-up issues. We have not received full confirmation as to what the underlying issue was, and will continue to look into a possible cause.