Is TeraSwitch down?

Teraswitch is looking into issues related to the R102 compute rack within TYO2.

A possible firmware/hardware bug was identified, and Teraswitch NOC elected to do a sequential software upgrade on the redundant switch pair in R102. After the reloads were complete, operations seemed to normalized. We will investigate this network switch pair further to understand the event better. Please report any outstanding issues with TYO2 to [email protected] or to our team directly.

Teraswitch is investigating reports of Internet connectivity loss at SGP2. We will provide an update shortly.

Teraswitch has isolated this incident to a vendor power maintenance event with unexpected impact, affecting both of our network racks and causing an Internet connectivity loss at SGP2. Customer services are believed to be powered on without upstream connectivity. Facility staff are at work to restore Internet connectivity and bring the site back online. We will provide an update as soon as possible.

Our facility partner has restored power to Teraswitch's SGP2 network racks. Root cause: both redundant PDUs in each network rack were fed by power breakout boxes connected to the same overhead electrical bus bar, creating a single point of failure. Facility engineers relocated one breakout box to the correct bus, restoring power to the network equipment and eliminating the single point of failure. We expect no further impact at this time. Customer systems did not lose power at any point during this event.

Teraswitch is investigating reports of intermittent network connectivity issues affecting some services in our EWR2 (Newark, NJ) location.

We have identified the issue affecting network connectivity in our EWR2 (Newark, NJ) location and are implementing corrective changes.

We have located an unstable core router in EWR2 and have implemented mitigations to divert traffic away from it.

Teraswitch monitoring reports that all impacts are currently resolved. Please report any further issues to our team. At EWR2, a device sitting between our data center fabric and our internet edge routers experienced a hardware fault that triggered an ASIC-level reload. After the reload, the device came back online in a degraded state and was unable to forward traffic from the data center fabric to the internet edge routers (specifically, it could not process VXLAN-routed traffic). EWR2 operates six of these core routers. Five were unaffected; however, the impaired unit failed to signal that it was incapable of forwarding traffic. Because downstream routing protocols did not detect this bad state, traffic that landed on the affected device was blackholed. Our team manually removed the router from the pool of forwarding destinations for this traffic, restoring service. We will perform all necessary follow-up steps to fully resolve the underlying issue and prevent recurrence.

Teraswitch identified a brief impact to network connectivity at our DUB1 and DUB2 sites beginning at 23:43 UTC. Undersea cables from both London and Amsterdam were lost, resulting in the brief loss of backbone and Internet connectivity at DUB1 and DUB2. Traffic was quickly restored via Internet routes and backbone connectivity via London followed shortly thereafter. Repair of the Amsterdam cable is also underway. Teraswitch is monitoring to confirm stability at these sites and will provide an update in the event of any further issues.

The Concerto cable was fully restored - multiple redundant links are now back online.

Teraswitch has identified a loss of two undersea cables landing in our SEA1 location from Tokyo. This is likely a terrestrial issue inside Seattle. Traffic was automatically diverted to our LA to Tokyo path.

Tokyo to Seattle has been nominal operation for many hours now. We are working with our vendors to understand the issue, but at this time we consider the incident and impact resolved.