Kalix EMR experienced a minor incident on October 7, 2018 affecting Kalix Platform and Messaging and 1 more component, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Oct 07, 2018, 12:42 PM UTC
We are migrating some data (reminders) and the move has taken longer than expected. Kalix is mostly usable but since the migration involves the indexing of records certain components are affected. For example if you create a new client or contact, the new record will not be searchable until the maintenance is complete. Or if you send a message, it will be queued until maintenance is complete. We expect that the maintenance will be complete by 1pm PST.
- monitoring Oct 07, 2018, 08:07 PM UTC
Unfortunately the estimated timeframe for the completion of the maintenance was underestimated. We have made some improvements to the indexing so it should complete faster, but there is still a little over half of the remaining data to go. After looking at how much data has been process, vs how much left to go, our updated maintenance time should be complete by 1am PST.
- monitoring Oct 08, 2018, 10:45 AM UTC
Unfortunately, the estimated completion time has been extended. We now calculate the maintenance will be completed by 2 pm PST Monday, October 8. We are still investigating ways of making the maintenance run faster. Kalix is functioning as normal with the exception of: The indexing of new clients and contacts (new clients and contact appearing in searches) The sending and receiving of messages e.g. appointment reminders, client responses The sharing of documents e.g. online forms, faxes The receiving of new notifications e.g. new appointment notifications, new message notifications The creation of back-ups The creation of insurance bill batches. There is currently a two-hour delay in these tasks. The delay will continue to reduce over time as the maintenance proceeds.
- monitoring Oct 08, 2018, 08:11 PM UTC
Unfortunately, in our attempts to improve the speed of the re-indexing, we inadvertently caused some secondary issues that increased the total time of the work. The queue length is back on its way down now, and we will be measuring how it goes over the next hour to get an idea of what is left. We are just under a 1/4 of what we started with, to give an idea of how much is left to go. We are estimating a 10 hour completion time from now 12 midnight PST.
- monitoring Oct 09, 2018, 08:10 AM UTC
We have updated the platform so that the big stream of data has been pushed off to a background queue. This means that Kalix will function as expected while the queue is finishing up. However any data that was produced yesterday is also in this stream, so it may take a few hours for all the data to appear from yesterday. We will give a final notice once the background queue is completely finished.
- resolved Oct 09, 2018, 10:44 AM UTC
All message have finally been processed - all previously saved data should now be up to date. Kalix is functioning again at 100%.