Hint Health incident

Hint experiencing slow performance, unusual database load

Critical Resolved View vendor source →

Hint Health experienced a critical incident on June 27, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jun 27, 2022, 12:21 PM UTC
Resolved
Jun 27, 2022, 12:21 PM UTC
Duration
—
Detected by Pingoru
Jun 27, 2022, 12:21 PM UTC

Update timeline

  1. resolved Jun 27, 2022, 05:37 PM UTC

    Our database automatically updated it's query statistics over the weekend and appears to have gotten a bad sampling, causing our database to stop using an index for a critical query. This caused significantly higher than usual database load and general performance slowness beginning around 2:46am PST, and triggering a production alert at 5:21am PST. Our team responded to the notification and began taking steps to return performance, which started to show improved performance around 6:11am. We re-analyzed the table at 6:55am PST and performance returned to normal at 6:57am PST. We're still investigating how to tweak the statistical sampler to prevent this possibility in the future.