doxy.me experienced a major incident on October 25, 2024 affecting Doxy.me API, lasting 10h 52m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 25, 2024, 12:36 AM UTC
We’re aware that patients are not appearing in their provider’s waiting room queue, and we are actively investigating the issue. We will provide updates as soon as we have more information. Thank you for your patience.
- monitoring Oct 25, 2024, 10:02 AM UTC
System is returning back to normal. We are continuing to monitor the situation.
- resolved Oct 25, 2024, 11:28 AM UTC
We recently encountered an issue that prevented patients from appearing in their provider’s waiting room queue. This issue has now been resolved, and patient check-in is functioning as expected. We will provide a post-mortem report in the coming days.
- postmortem Nov 04, 2024, 02:08 PM UTC
**Summary of event**: On October 24, patients who successfully checked in did not appear in the Waiting Room queue due to an incident from our real-time partner, PubNub. PubNub provides the technology that enables instant updates to [doxy.me](http://doxy.me) when a patient has joined a provider's Waiting Room and allows us to start a video call. This led to an outage on [doxy.me](http://doxy.me) approximately between October 24, 7:15 AM ET - 10:00 AM ET and October 24, 8:15 PM ET - October 25, 6:00 AM ET. **Business impacts:** Although providers were able to log in, their Waiting Room queue remained empty, preventing a session from starting between the patient and provider. **Root cause:** Our real-time partner, PubNub, encountered an unusual surge in traffic from sources other than [doxy.me](http://doxy.me), that temporarily overwhelmed their system, preventing it from responding quickly enough. As a result, PubNub’s communications to [doxy.me](http://doxy.me) were interrupted, causing patients to not appear in the Waiting Room queue. **How was the issue resolved:** PubNub blocked the source of surging traffic and made changes to handle the increased demand. Once PubNub made this update, [doxy.me](http://doxy.me) was fully operational again. [Doxy.me](http://Doxy.me) did not make any changes. **Preventative next steps:** [Doxy.me](http://Doxy.me) will collaborate with PubNub to ensure PubNub can manage traffic surges from its other customers and maintain scalable infrastructure, minimizing any potential impact on [doxy.me](http://doxy.me) services and customers.