ADESTIS Cloud incident

AHS-6009 - Major network issue in datacenter NBG6-NN

ADESTIS Cloud experienced a critical incident on March 4, 2022 affecting ADESTIS Cloud Computing-Infrastructure (NBG6-NN) and ADESTIS Cloud Storage (NBG6-NN) and 1 more component, lasting 15h 36m. The incident has been resolved; the full update timeline is below.

Started: Mar 04, 2022, 10:24 PM UTC
Resolved: Mar 05, 2022, 02:00 PM UTC
Duration: 15h 36m
Detected by Pingoru: Mar 04, 2022, 10:24 PM UTC

Affected components

ADESTIS Cloud Computing-Infrastructure (NBG6-NN)ADESTIS Cloud Storage (NBG6-NN)L2 / ToR (NBG6-NN)

Update timeline

investigating Sep 23, 2022, 02:08 PM UTC

The monitoring reports a partial outage of the datacenter (NBG6-NN)
investigating Sep 23, 2022, 02:10 PM UTC

Network issue escalated to second level team
investigating Sep 23, 2022, 02:13 PM UTC

Addtional external network engineers involved
identified Sep 23, 2022, 02:16 PM UTC

A network loop causing BUM traffic utilize all ports could be identified as the source. The switch fabric was rebooted and brought back to operations. Begin of recovery actions for affected workload.
monitoring Sep 23, 2022, 02:18 PM UTC

Infrastructure was restored. Begin of support actions for affected customers.
resolved Sep 23, 2022, 02:20 PM UTC

This incident has been resolved.
postmortem Sep 23, 2022, 02:21 PM UTC

# Summary On Friday 2022-03-04 around 23:23 CET a network disruption occurred in datacenter nbg6-nn which caused availability problems. ADESTIS immediately started troubleshooting and were able to recover the infrastructure around 2022-03-05 02:45 CET. ADESTIS then recovered the customer workload. # Details \(The following times are in CET\) 2022-03-04 23:23:33 First alerts were detected by the monitoring system. Some Infrastructure components are flapping. 2022-03-04 23:35:00 Problem analyzed by the 24/7 technician. 2022-03-05 01:30:00 Technician arrived at the datacenter. 2022-03-05 01:45:33 Response Team is working to find the root cause of a loop on some VPC Links of a core switch. 2022-03-05 02:30:00 A Loop caused by a faulty device is detected and stopped. \(Some switches needed to be rebooted\). 2022-03-05 02:45:00 Network traffic and workloads are recovering to normal state. 2022-03-05 03:00:00 Response Team is working on recovering all customer workloads which were affected. 2022-03-05 04:30:00 All workloads are back online. The Response Team is fixing application dependencies. Immediately after the incident occurred, we involved an external network specialist in the case. When the loop was detected we decided together to reboot the devices because the loop protection seems to not work anymore. We rebooted the secondary controller first and made sure it was back online before doing the same on the primary one. The reboot of the primary device has caused an automated reboot of other nodes. After the devices were online again the recovery process started. # Remarks Customers with redundant datacenter setup were only affected on location nbg6-nn. # Analysis The analysis of the log files showed an access violation of memory on a data plane node. After the access violation the node started to duplicate packets on all interconnection ports. Unfortunately, it is not possible to say how it came to the access violation because there was no core dump available. It could be an undetected memory corruption or more likely a software bug. However, it is not a known bug of the current firmware version. Our plans are now to switch to a different technology for the affected devices. The migration must be planned thoroughly. If customers need to be involved, we will contact them and plan the necessary migration steps together.