Hint Health incident
Hint experiencing slow performance, unusual database load
Resolved after 2 hours 51 minutes, on June 27, 2022.
Our database automatically updated it's query statistics over the weekend and appears to have gotten a bad sampling, causing our database to stop using an index for a critical query. This caused significantly higher than usual database load and general performance slowness beginning around 2:46am PST, and triggering a production alert at 5:21am PST. Our team responded to the notification and began taking steps to return performance, which started to show improved performance around 6:11am. We re-analyzed the table at 6:55am PST and performance returned to normal at 6:57am PST. We're still investigating how to tweak the statistical sampler to prevent this possibility in the future.
Update timeline
- resolved Jun 27, 2022, 05:37 PM UTC
Our database automatically updated it's query statistics over the weekend and appears to have gotten a bad sampling, causing our database to stop using an index for a critical query. This caused significantly higher than usual database load and general performance slowness beginning around 2:46am PST, and triggering a production alert at 5:21am PST. Our team responded to the notification and began taking steps to return performance, which started to show improved performance around 6:11am. We re-analyzed the table at 6:55am PST and performance returned to normal at 6:57am PST. We're still investigating how to tweak the statistical sampler to prevent this possibility in the future.
Your product runs on more than Hint Health
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services