Healthise experienced a major incident on September 25, 2023 affecting Coach and Communicate and 1 more component, lasting 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 25, 2023, 10:32 AM UTC
Healthwise-hosted solutions are experiencing performance issues. Our Network Administrators and Engineers are working to fix the problem. We will post updates as we learn more.
- resolved Sep 25, 2023, 10:41 AM UTC
All performance issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available.
- postmortem Sep 28, 2023, 04:34 PM UTC
## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the service disruption of the Healthwise-hosted solutions on September 25, 2023. ## Event Description At 4:05 AM MST, on Monday, September 25, 2023, Healthwise administrators were alerted to a service disruption for its Healthwise-hosted applications. Healthwise found that a 3rd party network appliance was the cause of the disruption. The appliance suffered a disruption in its ability to process its license, which resulted in the appliance processing data below functional levels. At 4:42 AM MST, Healthwise was able to restore service by resolving the licensing error which increased the amount of traffic that the network appliance could support. Total time of the incident was 37 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: The network appliance couldn’t support the volume of network traffic. Infrastructure engineers were able to mitigate the service by resolving a licensing error. Once the error was corrected the appliance was able to process traffic at expected levels. ## Corrective Action Healthwise increased the amount of traffic supported by the appliance and is currently migrating to a scalable environment to meet increased user demands over its products/services.