Hint Health experienced a minor incident on December 7, 2021 affecting HintOS App and AWS EC2 (Virginia), lasting 12h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 07, 2021, 05:33 PM UTC
Syncing data from Hint to partners is currently backed up by several hours. The solution to this problem is to increase the number of Hint's servers, however AWS is experiencing an issue that is makes scaling operations unsafe. AWS has identified the problem and is working on a fix. We will be scaling up our servers as soon as it is safe for us to do so, which will alleviate the problem.
- identified Dec 07, 2021, 08:42 PM UTC
There's no update from AWS as we continue to wait on them to resolve the underlying issue. In the meantime it remains unsafe for us to make any changes or updates to our production systems, which means we're unable to resolve the delays in data syncing. Our partner integration syncing is currently delayed about 3 hours and 40 minutes. We'll update as soon as we have more information. Apologies and thank you all for your patience!
- identified Dec 07, 2021, 11:44 PM UTC
AWS has indicated that, while most systems have been recovered, there are still issues impacting the US-EAST-1 region, which is our primary region. Aptible has confirmed that error rates are now much lower but that US-EAST-1 has not yet fully recovered. AWS engineers are hard at work resolving the final issues, and you can follow their status here: https://status.aws.amazon.com/
- monitoring Dec 08, 2021, 12:15 AM UTC
Systems are once again operational and our servers are being scaled. The backlog of sync events should be processed shortly.
- resolved Dec 08, 2021, 05:35 AM UTC
This incident has been resolved.