Healthise experienced a notice incident on November 2, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Nov 13, 2023, 09:48 PM UTC
At 5:26 PM MST, on Thursday, November 02, 2023, a change was made to retire content metadata for the Healthwise Advise solution. At 5:31 PM MST, production alerts were sent when a monitor failed. Healthwise administrators investigated and found that some content had incorrectly been marked as retired. At 7:57 PM MST, a fix was applied to restore the content and the content index was rebuilt. This made the content available for the Healthwise Advise application, but not for the Healthwise Education Module. Further investigation found that a backend service needed an authorization update to access the retired content. At 9:30 PM MST, a fix was made to a backend service so that the restored content could be retrieved by the Healthwise Education Module. The total time of the incident was 4 hours and 4 minutes.
- postmortem Nov 13, 2023, 09:49 PM UTC
## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the missing content in the Healthwise® Advise™ solution on November 02, 2023. ## Event Description At 5:26 PM MST, on Thursday, November 02, 2023, a change was made to retire content metadata for the Healthwise Advise solution. At 5:31 PM MST, production alerts were sent when a monitor failed. Healthwise administrators investigated and found that some content had incorrectly been marked as retired. At 7:57 PM MST, a fix was applied to restore the content and the content index was rebuilt. This made the content available for the Healthwise Advise application, but not for the Healthwise Education Module. Further investigation found that a backend service needed an authorization update to access the retired content. At 9:30 PM MST, a fix was made to a backend service so that the restored content could be retrieved by the Healthwise Education Module. The total time of the incident was 4 hours and 4 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: The Healthwise Advise solution was not able to access some content when an update to retire metadata incorrectly retired content. Additionally, a backend service’s authorization method prevented it from accessing the restored content. ## Corrective Action Content access was restored when the affected content was unretired and the content index was rebuilt. Additionally, the authorization for a backend service was updated so that the affected content could be accessed by the Healthwise Education Module. Healthwise is committed to improving its testing when releasing metadata updates and standardizing how services are authorized to access content.