Intermittent Routing Issues

Update timeline

1. investigating May 29, 2024, 05:56 PM UTC

   We have been alerted to a brief network convergence event that lasted approximately 30 seconds at 11:53AM CST. We just detected another approximately 20 second event. We have reached out to datacenter networking team as well as investigating possible causes. Currently we see a route reconvergence event at 11:53AM CST with Level3.

2. investigating May 29, 2024, 06:20 PM UTC

   The datacenter team has identified this as an issue with the blended bandwidth upstream service provider. We have noticed connectivity drops over the last 30 minutes that are affecting users with the drops occurring for about 10-30 seconds at a time. We are still working with the NetOps team to get this resolved.

3. identified May 29, 2024, 06:49 PM UTC

   Datacenter NetOps has stated that they identified a potential cause of the issue and have implemented a fix. We are currently monitoring the connectivity to PHX and can confirm we have not seen any intermittent drops in traffic over the last 28 minutes since our last status update. We are still in communication with NetOps to see what the potential cause may have been that led to this problem and what fixes have been implemented thus far to resolve the issue.

4. monitoring May 29, 2024, 07:36 PM UTC

   Datacenter NetOps has confirmed that a DDOS attack was the culprit behind the degraded network performance. As it stands, the attack has been mitigated and the datacenter is continuing to monitor the network. Adeptcloud was not the target in this DDOS attack, however, we were affected as the attackers targeted an upstream part of the network which was also responsible for handling our traffic. This resulted in issues with our network stability. We will provide more updates throughout the day if we learn any more information or experience any additional issues.

5. resolved Jun 04, 2024, 07:42 PM UTC

   This incident has been resolved.