Healthise experienced a critical incident on December 18, 2023, lasting 1h 35m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Dec 18, 2023, 07:52 PM UTC
Healthwise-hosted solutions are experiencing performance issues. Our Network Administrators and Engineers are working to fix the problem. We will post updates as we learn more.
- monitoring Dec 18, 2023, 08:01 PM UTC
We have identified the issue and implemented a temporary fix. We will continue to monitor the situation as we work on a permanent fix. We will post an update when that is completed.
- resolved Dec 18, 2023, 09:28 PM UTC
All performance issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available.
- postmortem Dec 21, 2023, 11:01 PM UTC
## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the service disruption for the Healthwise® Advise™ Reporting and Analytics application on December 18, 2023. ## Event Description At 12:11 PM MST, on Monday, December 18, 2023, an update was made to the security controls for the Reporting and Analytics application. This update caused an authentication issue that prevented the display of reporting data. No reporting data was lost. Healthwise was alerted to the issue at 12:22 PM MST and restored service by 12:57 PM MST. The total time of the incident was 35 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: At 12:11 PM MST, on Monday, December 18, 2023, a Healthwise administrator enhanced the security controls for an application service account. Once this change took effect, the application’s service account was restricted from getting reporting data which caused it to error. At 12:22 PM MST, Healthwise monitors sent alerts that the Reporting and Analytics application was experiencing issues. The team updated the security controls for the application service account and Healthwise production monitors reported that Reporting and Analytics was back up and running correctly at 12:57 PM MST. ## Corrective Action Access to Reporting and Analytics was restored when Healthwise Administrators updated the security controls for an application’s service account. We are actively working to improve communication so that the consequence of access control changes are broadly understood and approved by subject matter experts before they are changed.