ADESTIS Cloud experienced a minor incident on February 10, 2022 affecting Peering (NBG6-NN) and L2 / ToR (NBG6-NN), lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 23, 2022, 01:20 PM UTC
On 2022-02-10 we were informed by the monitoring about network issues. There was no clear source of the problem. These problems seemed to be temporary and automatically resolved after a few seconds.
- investigating Sep 23, 2022, 01:24 PM UTC
ADESTIS was checking all systems when another issues was reported. But since the problem was gone after a few seconds there was not enough time to find the source of the problem. Problems reported on 09:55, 11:22, 11:43 We could track down the problem to a switch fabric with six members and we decided to restart the backup routing engine of the fabric.
- investigating Sep 23, 2022, 01:26 PM UTC
The problem occured again at 15:06.
- investigating Sep 23, 2022, 01:28 PM UTC
After further checks and preparations we also decided to restart the master routing engine. (18:20) The problem seems to be gone.
- investigating Sep 23, 2022, 01:30 PM UTC
The problem came back and we created a ticket with the supplier
- investigating Sep 23, 2022, 01:31 PM UTC
After hours of troubleshooting together with the supplier no source could be identified.
- identified Sep 23, 2022, 01:40 PM UTC
We were able to identify the issue. We were able to track down a flood of BUM traffic originating to the infrastructure of the N-IX internet exchange point. After disabling the port the problem stopped immediately.
- resolved Sep 23, 2022, 01:41 PM UTC
This incident has been resolved.
- postmortem Sep 23, 2022, 01:42 PM UTC
We have received large amounts of BUM traffic via the N-IX. A packet capture shows that another customer of the N-IX internet exchange was affected by a DDoS attack and due to a software error on their equipment that traffic was flooded to us. In the past the port for the N-IX was terminating on switch infrastructure because of private VLAN usage. In order to not run into similar problems we have reworked our network infrastructure since then so that external traffic can only reach us via L3 to prevent similar problems.