Doxy.me Service Interruption

Update timeline

1. investigating May 25, 2023, 06:29 PM UTC

   We are aware of an outage currently affecting doxy.me. This appears to affect sign-in, call connecting, and various other system services. Our team is currently investigating at this time. We will post an update as we know more.

2. monitoring May 25, 2023, 07:06 PM UTC

   We are seeing our system stabilizing. We are continuing to look into the incident and are going to continue to monitor the situation.

3. monitoring May 26, 2023, 01:00 PM UTC

   System levels have remained stable overnight. We are continuing to monitor the system performance and will provide another update by 12pm ET.

4. resolved May 26, 2023, 04:01 PM UTC

   Doxy.me services continue to be stable. We are considering this incident resolved and will provide a post mortem within the few business days.

5. postmortem Jun 02, 2023, 05:35 PM UTC

   **Summary of event:** On May 25th, [Doxy.me](http://doxy.me/) experienced a database issue that resulted in 27 minutes of unplanned downtime from 2:03pm to 2:30pm ET. **Business Impacts:** Sessions in progress were not impacted. The incident resulted in providers being unable to log in for their sessions. In addition, patients who were not already in the queue were unable to check in to their providers waiting room. **Root cause:** A data import file was processed internally. While it was being processed, a series of long-running secondary server processes were triggered. Unfortunately, these processes built up in number, consuming additional server resources until the servers were backed up. **How was the issue resolved:** The file was able to finish processing and the system self recovered. **Preventative next steps:** We are optimizing the the long-running processes which will help prevent this scenario from happening again. In the short term, we are no longer running large data imports during peak business hours.