ADESTIS Cloud experienced a minor incident on July 27, 2024 affecting Backbone (FFM1-NTT) and Backbone (NBG6-NN) and 1 more component, lasting 1h 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 27, 2024, 05:17 PM UTC
We are currently investigating this issue.
- monitoring Jul 27, 2024, 05:39 PM UTC
A malformed BGP packet was received by our edge routers. There was a short interruption to the internet on one location and a redundancy loss on another location. All sessions were restored and we are currently monitoring the systems.
- resolved Jul 27, 2024, 06:51 PM UTC
We have opened a ticket with the device manufacturer and check for further actions in the next days.
- postmortem Jul 30, 2024, 10:42 AM UTC
A part of our routers were affected by malformed BGP packets. Taken measurements \(BGP Error Tolerance\) which should prevent BGP problems had no effect. When a malformed BGP update was received there was a rare situation where an assert in the router software was triggered and the routing process was restarted. After analyzing the issue we properly hit a bug in the router software and we needed to upgrade all routers to a version were the problem was fixed.