Healthise experienced a critical incident on August 25, 2022 affecting Coach, lasting 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 25, 2022, 02:16 PM UTC
The Healthwise-hosted instance of Coach is currently experiencing an outage due to spikes in network traffic. We are working to fix the problem and will post updates as we learn more.
- resolved Aug 25, 2022, 02:47 PM UTC
All issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available.
- postmortem Sep 14, 2022, 10:47 PM UTC
# **Event Description** At 03:07 AM \(MST\) on Thursday, August 25, 2022, Healthwise administrators were made aware of intermittent service degradation of the Healthwise Coach application and began identifying the causes of the service degradation. Our development and operations team deployed a long-term fix on 8/25/2022 for Healthwise Coach to reduce the risk of similar incidents in the future. The total time of degradation was 5 hours and 25 minutes. # **Findings and Root Cause** Based on the investigation conducted, the team determined the following findings regarding this event: A Healthwise customer was performing automated testing against our production environment, which included a high volume of unauthorized requests to the Coach application. We isolated the requests to this single customer and found that the Coach application incorrectly responded to this type of request with multiple redirects, which eventually overwhelmed the servers. # **Corrective Action** Healthwise updated the Coach application to send only a single redirect response when an unauthorized request is made. Healthwise deployed the update on 8/25/2022.