TeraSwitch incident

EWR1 - Network backbone traffic instability

Notice Resolved View vendor source →

TeraSwitch experienced a notice incident on February 26, 2025 affecting EWR1 - Newark, NJ and DAL1 - Dallas, TX and 1 more component, lasting 21h 26m. The incident has been resolved; the full update timeline is below.

Started
Feb 26, 2025, 05:22 PM UTC
Resolved
Feb 27, 2025, 02:49 PM UTC
Duration
21h 26m
Detected by Pingoru
Feb 26, 2025, 05:22 PM UTC

Affected components

EWR1 - Newark, NJDAL1 - Dallas, TXAMS1 - Amsterdam, NetherlandsDUB1 - Dublin, Ireland

Update timeline

  1. investigating Feb 26, 2025, 05:22 PM UTC

    Teraswitch is investigating backbone instability on certain destinations that passed through our EWR1 location in US East.

  2. resolved Feb 27, 2025, 02:49 PM UTC

    After a lengthly review of our backbone operations in EU, a misconfiguration was located that may have caused a sub-set of global traffic to detour through Frankfurt that should not have. The impact was determined to be relatively limited in scope (~20gbps of traffic.) Customer reports of issues were zero, but our monitoring leads us to believe some specific destinations Dallas/Newark -> Dublin, Amsterday -> Dallas/Newark should have seen packet loss in one direction. Network engineering will continue to investigate this issue internally to improve the system that allowed this mistake to be unnoticed, but this specific impact should be considered resolved.