Clinical Advise Authentication Issues for Redox Customers

Update timeline

1. investigating Oct 03, 2023, 02:38 PM UTC

   We are currently investigating an issue affecting Advise clients when they try to authenticate to the application.

2. monitoring Oct 03, 2023, 03:21 PM UTC

   We have identified the issue and implemented a temporary fix. We will continue to monitor the situation as we work on a permanent fix. We will post an update when that is completed.

3. monitoring Oct 03, 2023, 04:02 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Oct 03, 2023, 04:02 PM UTC

   All performance issues have been resolved. We will post a root cause analysis once we have completed our full investigation. If the investigation has not been completed within 1 week we will post an interim RCA with the information that we currently have available.

5. postmortem Oct 11, 2023, 08:21 PM UTC

   ## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the service disruption of the Healthwise-hosted solution between October 2nd and October 3rd, 2023. ## Event Description At 5:30 PM MST, on Monday, October 02, 2023, an update was made to the Healthwise® Advise integration. This update caused authentication issues for the Healthwise Advise application. Healthwise was alerted to the issue at 7:38AM MST and restored service by 9:20 AM MST. The total time of the incident was 15 hours and 50 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: Healthwise was contacted by a 3rd party vendor regarding recommended security enhancements for a service update. The updates were for the FHIR API and how Healthwise applications authenticate to it. A developer implemented the enhancements to the Advise solution at 5:30 PM MST on October 2nd. Healthwise SLA monitors didn’t detect the authentication issue and Healthwise immediately responded when a disruption of service was reported at 7:38 AM MST on Tuesday, October 3rd. Engineers were able to restore service at 9:20 AM MST by reconfiguring how the application authenticates. ## Corrective Action Healthwise updated an authentication setting to complete the security update and restore the applications ability to authenticate. This incident did not trigger the SLA monitors. We are actively working to ensure we are notified of future authentication issues and improve the timeliness of our response.