dataforest experienced a minor incident on August 5, 2025 affecting Datacenter Core Routing & Switching, lasting 2h 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 05, 2025, 10:59 AM UTC
Our monitoring systems detected short periods of packet loss to some destinations within our maincubes FRA01 location at 11:47:42, 12:15:34, 12:18:02, and 12:24:23 CEST, each lasting around 15 seconds. While the situation has been stable since then, we're still looking into the root cause. Initial findings point to log messages on one of the two redundant core switches involved, while the other didn't log anything unusual. This suggests that the impact for customers was likely minimal, but we're continuing our investigation and will keep the incident open until it's fully clarified and mitigated.
- identified Aug 05, 2025, 12:24 PM UTC
We have identified the root cause and will carry out an emergency configuration change in the next 60 minutes. This may result in a brief period of packet loss but is necessary to ensure stable operation.
- resolved Aug 05, 2025, 01:58 PM UTC
The configuration change was successfully applied with minimal impact at 15:18 / 15:27 CEST. The root cause was an inconsistent Spanning Tree configuration within our maincubes FRA01 core network, which has now been resolved. Only a limited number of racks were affected, depending on the specific configuration of their Top-of-Rack switches. Over the coming weeks and months, we will review the configuration of all devices involved in the topology to identify potential improvements. Should further changes be necessary, we will implement them during announced maintenance windows at night, as the immediate issue has been resolved and no further disruptions are expected.