Healthise experienced a major incident on December 15, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Dec 18, 2020, 11:25 PM UTC
All performance issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available
- postmortem Dec 18, 2020, 11:27 PM UTC
The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the performance degradation and intermittent outages of the Healthwise Coach, EMR Module, and Custom Content manager applications on December 15, 2020. # Event Description Beginning at 08:30AM MST on Tuesday, December 15, 2020, the Healthwise Coach, EMR Module, and Custom Content Manager applications experienced degradation significant enough to impact the client experience. Healthwise identified a resource issue on one of the web servers hosting the Healthwise applications. Healthwise removed the affected server from the load balancer and rebooted to terminate the affected process. Resource usage returned to normal levels at 09:09AM MST. The approximate length of the degradation was 39 minutes. During that time, applications were unavailable for approximately 16 minutes. # Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: A process on the Healthwise Custom Content Manager application was consuming all the available resources on one of the web servers responsible for hosting the Healthwise applications. This, combined with an increased usage of Healthwise's Redis session cache, caused the application degradation until that server was removed from the load balancer and rebooted to terminate the abnormal instance of the process. # Corrective Action Healthwise rebooted the affected server and is continuing to monitor resource usage across the Healthwise application suite. Healthwise also continues to evaluate its use of the Redis cache for sessions to ensure resources are freed appropriately.