Coach, EMR, and Custom Content Manager partial outage

Update timeline

1. investigating Oct 14, 2021, 08:48 PM UTC

   We are currently investigating an issue with our systems affecting the ability to search for and retrieve content. Only some clients are affected.

2. resolved Oct 14, 2021, 09:04 PM UTC

   All performance issues have been resolved.

3. postmortem Oct 21, 2021, 02:41 PM UTC

   # Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the performance degradation and intermittent outages of the Healthwise Coach, EMR Module, and Custom Content Manager applications on October 14, 2021. # Event Description At 02:17PM MST on Thursday, October 14, 2021, Healthwise administrators were made aware of the EMR Module application experiencing degradation and intermittent availability. Healthwise identified that the search indices on one of the application servers was corrupted while rebooting as a result of the application of routine Windows Updates. Healthwise removed the affected server from the load balancer and re-built the search indices on the affected server. After these steps were completed, the server was validated and restored to the load balancer. Degradation and availability issues were resolved at 02:48 PM MST. The approximate length of the degradation was 31 minutes. # Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: A reboot following the routine application of a Windows Update impacted the search indices on an individual application server. This resulted in affected applications intermittently failing, if they used the impacted server, until the server was removed from the load balancer and the search indices were rebuilt. # Corrective Action Healthwise is reviewing its automated Windows Updates processes and will implement additional checks in order to ensure updates complete as expected.