Healthise experienced a notice incident on June 15, 2023 affecting Coach and Communicate and 1 more component, lasting 2h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 15, 2023, 03:56 PM UTC
Healthwise-hosted solutions are experiencing performance issues. Our Network Administrators and Engineers are working to fix the problem. We will post updates as we learn more.
- investigating Jun 15, 2023, 04:42 PM UTC
We are continuing to investigate this issue.
- monitoring Jun 15, 2023, 04:56 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 15, 2023, 06:15 PM UTC
This incident has been resolved.
- postmortem Jun 23, 2023, 06:44 PM UTC
## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the performance issues of the Healthwise-hosted solutions on June 15, 2023. ## Event Description At 09:35 a.m. MST on Thursday, June 15, 2022, Healthwise administrators were alerted to product errors from Healthwise-hosted applications. Healthwise found that the connection limit of a shared database pool had been reached and no further connections were allowed. At 10:51 a.m. MST Healthwise was able to restore service by dropping connections and rebuilding a database index. Total time of the incident was 1 hours and 16 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: The root cause was a scheduled back-end maintenance task that caused high latency to a critical back-end service used for authentication. As the critical back-end service became degraded this caused a disruption to all dependent systems ## Corrective Action Healthwise administrators dropped database connections so the database index could rebuild. Once the index was built, the database server was able to fulfill requests faster than new requests were created. Healthwise will follow database administration procedures to ensure index' are available when needed.