doxy.me experienced a minor incident on February 10, 2021 affecting Doxy.me webpages, lasting 18h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 10, 2021, 04:56 PM UTC
We are currently investigating latency spikes that are preventing patients from checking in.
- investigating Feb 10, 2021, 05:25 PM UTC
We are continuing to investigate this issue. We are seeing patients are now able to check in, but are not showing up for the provider queue.
- monitoring Feb 10, 2021, 06:26 PM UTC
We are continuing to monitor the latency issues with our check-in vendor as their statistics are trending in the right direction. Internally, we are intermittently able to check in and have calls, but we are also still seeing the server errors. We do not have an ETA for a fix to share at this time. We will provide another update in 20 minutes.
- monitoring Feb 10, 2021, 06:47 PM UTC
The vendor is reporting that their main functionality has been restored. We are still monitoring the API calls as they are continuing to recover from today's event. These generally take more time to return to normal. We will provide another update in 20 minutes.
- monitoring Feb 10, 2021, 07:12 PM UTC
We are continuing to monitor the incident. We are still seeing improvements as the vendor's system continues to recover. We will provide another update in 20 minutes.
- monitoring Feb 10, 2021, 07:33 PM UTC
The vendor is only seeing a few remaining service errors being logged on their side. They are continuing to address the elevated latency issues while we continue to monitor our API calls.
- monitoring Feb 10, 2021, 08:00 PM UTC
We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.
- monitoring Feb 10, 2021, 08:27 PM UTC
We are continuing to monitor this incident and are still seeing intermittent errors due to latency. Overall, the system continues to show a much quicker response time than it did earlier in the day. The vendor is continuing to work on fixes to bring everything back to an operational status.
- monitoring Feb 10, 2021, 08:48 PM UTC
We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.
- monitoring Feb 10, 2021, 09:32 PM UTC
We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.
- monitoring Feb 10, 2021, 10:14 PM UTC
The vendor has implemented a fix to route traffic around the failure in their system as a temporary fix. In addition, this incident appears to have been magnified by an incident at Amazon Web Services (AWS). AWS is currently rolling out a fix on their side to address problems with their Network Load Balancers. We are continuing to monitor this incident and are seeing a significant reduction in the errors being logged since these efforts have been announced.
- monitoring Feb 10, 2021, 11:25 PM UTC
The vendor is continuing to monitor their system and has updated their status back to fully operational. AWS is continuing to roll out their updates to resolve the issue with their network load balancers. Our logged errors have been drastically reduced thanks to these efforts. We will continue to monitor this incident into tomorrow.
- resolved Feb 11, 2021, 01:34 PM UTC
AWS has resolved the issue with the network load balancers and is showing their service is operating normally. We are no longer seeing latency spikes. This issue is now resolved.