TeraSwitch Outage History

# Root Cause Analysis: AMS1 Connectivity Outage **Incident Date:** March 26, 2025 **Duration:** ~1h 21m \(22:08 UTC – 00:14 UTC \+1\) **Severity:** Critical – Full site connectivity loss **Affected Site:** AMS1 – Amsterdam, Netherlands **Status:** Resolved ## Executive Summary On March 26, 2025, Teraswitch's AMS1 facility experienced a complete loss of backbone connectivity lasting approximately 1 hour and 21 minutes. The outage was caused by a scheduled fiber vendor maintenance window that simultaneously impacted both the primary and what was believed to be a diverse redundant fiber path between AMS1 and the rest of the backbone. Investigation revealed that due to a documentation and handoff error at the fiber vendor dating back over a year, the AMS1–AMS2 fiber span had never been migrated to the intended diverse path as part of the AMS3 ring buildout. As a result, both affected spans shared physical infrastructure, eliminating the redundancy intended to protect against exactly this type of event. Connectivity was restored within the maintenance window after a rapid joint audit with the fiber vendor confirmed the provisioning discrepancy, and the AMS1–AMS2 span was moved to its correct diverse path. ## Background Teraswitch's Amsterdam backbone originally consisted of a single dark fiber span between AMS1 and AMS2. When AMS3 was later brought online, the network design called for a three-node fiber ring with fully diverse physical paths between all sites to provide redundant backbone connectivity. To support this design, the fiber vendor was engaged to: 1. Provision a new AMS1–AMS3 span 2. Provision a new AMS2–AMS3 span 3. Reroute the existing AMS1–AMS2 span onto a physically diverse path Due to an internal handoff and documentation error within the fiber vendor, step 3 was not completed. The AMS1–AMS2 span remained on its original physical route. Teraswitch was not made aware of this omission, and the span continued to operate normally for over a year. Because it carried live traffic and appeared correctly in topology, it was not identified as incorrectly provisioned during subsequent audits. ## Timeline of Events | Time \(UTC\) | Event | | --- | --- | | Prior to March 26 | Fiber vendor schedules routine maintenance affecting Amsterdam infrastructure | | ~22:08 | AMS1 backbone connectivity lost. Both AMS1–AMS2 and AMS1–AMS3 paths go down simultaneously. Teraswitch NOC begins triage. | | ~22:53 | Fiber vendor engaged and confirms maintenance is impacting both spans. Root cause identified as shared physical infrastructure due to the original provisioning error. | | ~00:14 \+1 | Fiber vendor migrates AMS1–AMS2 span to the correct diverse physical path. Connectivity restored. Monitoring confirmed stable. | ## Root Cause **Primary cause:** An internal documentation and handoff failure at the fiber vendor resulted in the AMS1–AMS2 dark fiber span never being migrated to its intended physically diverse route during the AMS3 ring buildout. Both the AMS1–AMS2 and AMS1–AMS3 spans shared common physical infrastructure, making the designed ring topology's redundancy ineffective. **Contributing factor:** Because the span was operationally active and traffic was flowing normally, the provisioning error went undetected across both Teraswitch and vendor records for over a year. When the scheduled maintenance affected the shared physical infrastructure, both paths were impacted simultaneously, leaving AMS1 with no available backbone connectivity. ## Impact * **AMS1 customers** experienced a complete loss of inbound and outbound connectivity for approximately 1 hour and 21 minutes. * No data loss or hardware damage occurred. * All other Teraswitch sites were unaffected. ## Resolution Working jointly with the fiber vendor during the incident, Teraswitch engineers and vendor technicians audited the physical path assignments for all Amsterdam spans. The discrepancy between the intended and actual routing of the AMS1–AMS2 span was identified. The vendor migrated the span to the correct physically diverse path, restoring independent redundant connectivity across the AMS1–AMS2–AMS3 ring as originally designed. ## Corrective Actions | Action | Owner | Status | | --- | --- | --- | | Confirm and document physical path diversity for all three Amsterdam spans with fiber vendor | Teraswitch / Fiber Vendor | Complete | | Obtain updated as-built fiber records from vendor reflecting correct path assignments | Fiber Vendor | Complete | | Audit all other Teraswitch sites for similar provisioning discrepancies against vendor records | Teraswitch | Complete | | Establish a fiber path verification checklist for all future vendor provisioning work prior to accepting new spans | Teraswitch | Planned | | Add physical diversity validation to change management process for any future ring or redundancy buildouts | Teraswitch | Planned | | Incorporate optical span latency validation against fiber path build sheets as an acceptance criterion for new span provisioning | Teraswitch | Planned | ## Lessons Learned * **Operational traffic is not proof of correct provisioning.** A span can carry live traffic for an extended period while still being routed incorrectly relative to its intended physical diversity design. * **Redundancy assumptions must be periodically verified against vendor as-built records**, not solely inferred from operational status. * **Fiber vendor handoffs require explicit acceptance criteria** including documented physical path confirmation before provisioning work is considered complete. * **Span latency is a low-cost signal for path verification.** In post-incident review, Teraswitch noted that the measured propagation latency on the AMS1–AMS2 span was slightly lower than expected based on the fiber path build sheet for the intended diverse route. This discrepancy, while subtle, was consistent with the span still traversing the shorter original path. Validating measured latency against estimated values from build sheets at provisioning acceptance could have surfaced this error significantly earlier. This check will be incorporated into the span acceptance process going forward. _RCA prepared by Teraswitch Network Engineering. For questions contact the NOC or network architecture team._