Healthise experienced a minor incident on January 25, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 25, 2024, 10:32 PM UTC
At 9:40 AM MST, on Saturday, January 13, 2024, Healthwise administrators were alerted to an issue with the Fulfillment Service and began an investigation. They found that a background database process was creating slower than expected response times for requests to the fulfillment service. The response times were slower than the threshold for the SLA monitors. Healthwise administrators stopped a background database process which improved resource availability and restored service availability. Production monitors reported stable performance at 10:45 AM MST. The total time of the incident was 1 hour and 5 minutes.
- postmortem Jan 25, 2024, 10:33 PM UTC
## Introduction The purpose of this Root Cause Analysis \(RCA\) is to determine the causes that contributed to the intermittent degradation of the Fulfillment Service on January 17, 2024. ## Event Description At 9:40 AM MST, on Saturday, January 13, 2024, Healthwise administrators were alerted to an issue with the Fulfillment Service and began an investigation. They found that a background database process was creating slower than expected response times for requests to the fulfillment service. The response times were slower than the threshold for the SLA monitors. Healthwise administrators stopped a background database process which improved resource availability and restored service availability. Production monitors reported stable performance at 10:45 AM MST. The total time of the incident was 1 hour and 5 minutes. ## Findings and Root Cause Based on the investigation conducted, the team determined the following finding regarding this event: A background database process was using a lot of server resources. When another high resource process was initiated the fulfillment service response time exceeded the monitors wait time. A few requests to the fulfillment service timed out, but the vast majority of them were processed successfully. ## Corrective Action Performance was restored when the background database process was stopped. Healthwise administrators scaled server resources and will monitor consumption while the background task is completed. We will will also audit the high resource process for opportunities to improve its efficiency.