doxy.me experienced a major incident on March 10, 2022 affecting Doxy.me API, lasting 7h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 10, 2022, 03:11 PM UTC
We are currently investigating the issue.
- investigating Mar 10, 2022, 03:11 PM UTC
We are continuing to investigate this issue.
- identified Mar 10, 2022, 03:20 PM UTC
The issue has been identified, and we are currently working on a fix.
- monitoring Mar 10, 2022, 03:49 PM UTC
A fix has been implemented and we are continuing to monitor the situation.
- resolved Mar 10, 2022, 10:35 PM UTC
The fix we implemented has corrected the problem. We are marking this incident resolved. We will be sharing a post mortem in the coming days.
- postmortem Mar 14, 2022, 02:48 PM UTC
**Summary of event:** [Doxy.me](http://Doxy.me) experienced a severe increase in API response time. **Business Impacts:** [Doxy.me](http://Doxy.me) was unresponsive from 10:01 AM until roughly 10:20 AM. **How was the issue resolved:** We rolled back an update that was pushed out this morning. Removing the update from our production servers allowed our servers to recover and return to normal capacity. **Root cause:** One of the updates we rolled out on 3/10/2022 caused a problem with our servers writing to the cache. An error occurred as we were writing the data to the cache, which led to the servers failing. As servers failed, response times increased and the system became unresponsive. **Preventative next steps:** We have prepared a fix to the problematic code and will be redeploying the update with the fix next week. In addition, we are improving our testing on our staging environments to help expose these problems before they are introduced to our providers and patients.