Healthise experienced a major incident on February 19, 2020 affecting EMR Modules, lasting 2h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 19, 2020, 05:56 PM UTC
Healthwise-hosted solutions are experiencing performance issues. Our Network Administrators and Engineers are working to fix the problem. We will post updates as we learn more.
- monitoring Feb 19, 2020, 07:29 PM UTC
We have identified the issue and implemented a temporary fix. We will continue to monitor the situation as we work on a permanent fix. We will post an update when that is completed.
- resolved Feb 19, 2020, 08:01 PM UTC
All performance issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available.
- postmortem Feb 24, 2020, 11:14 PM UTC
The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the performance degradation and intermittent outages of the Healthwise EMR Module on February 19, 2020. # Event Description Beginning at 10:59AM MST on Tuesday, February 18, 2020, Healthwise received reports of intermittent failures of the EMR Module application. These failures subsided throughout the day. On Wednesday, February 19, 2020, these intermittent failures returned. Starting at 10:03 AM MST, the application became unresponsive. After determining the issue to be caused by high CPU utilization on one of the databases used by the application, Healthwise deployed additional resources to the database. Healthwise also identified that recent application changes contributed to the problem and deployed a code change to address the issue. Services were restored at 12:29 PM MST. The approximate length of the degradation was 2 hours and 26 minutes. # Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: On Monday, February 17, 2020, Healthwise deployed EMR Module UI enhancements that introduced code that was not performant under load. The introduction of this code gradually led to 100% utilization of a database critical to the app, causing slow response time, and at times, making the application completely unavailable. # Corrective Action Healthwise deployed additional CPU resources to the affected database and completed and out-of-cycle deployment of the application to improve the performance. Healthwise development teams will continue investigating further performance improvements.