Adeptcore experienced a critical incident on November 6, 2022 affecting ACP - Network, lasting 1d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 06, 2022, 11:47 AM UTC
We are investigating reports of a public WAN interruption. At this time we have confirmed that all equipment is online and we are working to resolve the WAN issues.
- identified Nov 06, 2022, 11:56 AM UTC
We have identified the issue to be packet loss we are seeing an unusually high number of packets headed to us being dropped. Datacenter NOC sees the same issue and is working on resolution of the network. We have confirmed all customer VM's, services, storage, address space and power are all online but may be experiencing packet loss when trying to reach out IP Space.
- identified Nov 06, 2022, 12:12 PM UTC
The datacenter networking team is seeing routing related issues when going through Level3. Testing with Windstream has shown no packet loss occurring. As such, routing changes are currently being implemented at the PHX datacenter to eliminate Level3. We will provide an additional update once the routing changes have been confirmed.
- monitoring Nov 06, 2022, 12:56 PM UTC
The datacenter has made peering changes to adjust the broken routes and we are no longer seeing packet loss to our IP space. From the looks of it, the last five minutes have been steady in regards to latency and packet loss. We are continuing to monitor this. If clients are using ScreenConnect or RMM solutions, we are seeing most of those services starting to come back online. We will provide an update if we detect any more issues with the network throughout the day or when we find out exact cause of Level 3 Networks/CenturyLink issues today.
- resolved Nov 07, 2022, 02:48 PM UTC
Sunday morning around 5 AM CST, our Phoenix data center's Lumen network links experienced a 300Gbps DDOS (Distributed Denial-of-service attack). The affected network was re-routed and the datacenter's DDOS mitigation and scrubbing services began clearing the traffic. Between 5 AM and 11 AM CST, additional DDOS attacks were mitigated at which point a small amount of packet loss was experienced. Packet loss and network stability was 100% since 11 AM and has been throughout the day into Monday morning (today). We are continuing to monitor this situation but have not seen any service disruptions since the aforementioned time frame and will be marking this incident as resolved. An RFO will be provided within the next seven days in regards to this issue.