Kalix EMR experienced a minor incident on November 28, 2018 affecting Kalix Platform and Online Schedulers, lasting 13d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 28, 2018, 01:59 PM UTC
A few times a day the Kalix servers are 'spiking' causing a slowdown. We have been releasing updates to fix the issue over the past week. Unfortunately, the problem is still occurring. We are continuing to investigate the cause with the aim to resolve ASAP. The best way to recover when this issue occurs is to refresh the page (Command/Control R)
- identified Nov 29, 2018, 02:09 PM UTC
We released an update to improve this issue. We are currently working on more permanent solution, by modifying the way Kalix is storing data.
- identified Nov 30, 2018, 04:25 PM UTC
We have pushed up the first update toward migrating data to somewhere faster. We have noticed some improvement. We will continue this process. It may probably take another week to complete. There will be ongoing updates on this.
- identified Dec 04, 2018, 01:33 PM UTC
Our first migration of data is complete (client records). We are about to release an update to make this change live.
- monitoring Dec 10, 2018, 12:58 PM UTC
We have released the update that uses the migrated client records. This should significantly reduce the load on our main servers and hopefully reduce the server spikes that we have been seeing. As a side effect client searches should be much quicker and more consistent. Things were looking stable over the weekend, but we will be monitoring the performance of the server today to see if any more action is required. Hopefully this issue can be put to rest.
- resolved Dec 11, 2018, 05:00 PM UTC
We feel that the client update has improved the server enough for us to close this issue for now. At some point in the future we will be returning to this work to speed up other parts of Kalix, but for now we will be focusing on finishing our updates to telehealth and payments.