A service degradation in search service has been detected our engineers are investigating it.

Update timeline

1. investigating Sep 01, 2023, 12:51 AM UTC

   We are currently investigating this issue.

2. investigating Sep 01, 2023, 01:15 AM UTC

   Service has been restored. We will follow up with a root cause analysis shortly.

3. resolved Sep 01, 2023, 01:15 AM UTC

   This incident has been resolved.

4. postmortem Oct 17, 2023, 03:55 PM UTC

   ## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to issues with accessing the Healthwise Search Service on August 31, 2023. ## Event Description At 6:42 PM MST on Thursday, August 31, 2023, Healthwise administrators were alerted that the Search Service was unavailable. Healthwise found the service was overwhelmed due to heavy network traffic. At 7:12 PM MST Healthwise was able to restore service by reducing the requests. Total time of the incident was 30 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following findings regarding this event: One client was generating more network traffic than the service could handle. Infrastructure engineers were able to restore the service by isolating the disruptive traffic. ## Corrective Action We’re actively implementing a solution to optimize the flow of requests to enhance system stability and fairly allocate resources. This will prevent resources from being monopolized and ensure they are available for all requests. We also disabled access for the keys that overwhelmed the service.