TeraSwitch Outage History

Started Dec 05, 2025, 09:53 PM UTC· resolved Dec 05, 2025, 11:46 PM UTC

1. investigating Dec 05, 2025, 09:53 PM UTC

   Teraswitch is investigating increased errors in our service portal (console.tsw.io). Teraswitch services themselves are unaffected. We will provide an update as more information becomes available.

2. resolved Dec 05, 2025, 11:46 PM UTC

   The cause of the increased errors was identified and fixes put in place. This issue is resolved.

Read the full incident report →

Started Nov 29, 2025, 05:21 AM UTC· resolved Nov 29, 2025, 01:02 PM UTC

1. monitoring Nov 29, 2025, 05:21 AM UTC

   Terawitch Network Operations has observed a subsea cable fault between our Seattle and Tokyo POPs. Latency to Tokyo may be increased while the system is down. We have reached out to our cable provider for an update.

2. resolved Nov 29, 2025, 01:02 PM UTC

   Fault has cleared and cable returned to service.

Read the full incident report →

Started Nov 18, 2025, 12:08 PM UTC· resolved Nov 18, 2025, 06:09 PM UTC

1. investigating Nov 18, 2025, 12:08 PM UTC

   Teraswitch is investigating monitoring alerts/errors in our console (console.tsw.io) and API (api.tsw.io). This appears to be due to an underlying major cloud provider which is experiencing an outage. We will provide an update as more information becomes available.

2. identified Nov 18, 2025, 12:15 PM UTC

   This issue has been isolated to an ongoing Cloudflare network outage - see their incident page for more details: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

3. resolved Nov 18, 2025, 06:09 PM UTC

   Cloudflare updated their incident report that their services are now operating normally - they will post a final update to the incident once their investigation has concluded. Teraswitch monitoring confirms our console / API services have stabilized with no further errors. This incident is now resolved.

Read the full incident report →

Started Nov 14, 2025, 03:17 PM UTC· resolved Nov 17, 2025, 01:45 PM UTC

1. investigating Nov 14, 2025, 03:17 PM UTC

   Teraswitch is investigating moments of Internet connectivity instability.

2. monitoring Nov 14, 2025, 05:25 PM UTC

   Teraswitch was attacked by a DDOS attack that specifically targeted our infrastructure and blockchain services. While our network has the capacity and mitigation ability, it took longer than expected to identify good traffic from attack traffic without further causing worse issues. Some attack traffic sources came from very abnormal and specific locations, such as AT&T (US ISP) and a group of Brazilian ISPs. This traffic primarily landed in our Dallas data center, which overloaded select network links there on its way towards the attack targets. We have tightened our rule sets, shifted traffic around, and are working to ensure impacts are continually mitigated.

3. identified Nov 15, 2025, 01:29 PM UTC

   We are observing further Internet connectivity instability due to the attack and are continuing to apply mitigations. We will continue to monitor the situation until resolution and update on any further changes.

4. resolved Nov 17, 2025, 01:45 PM UTC

   This incident is now considered resolved.

Read the full incident report →

Started Oct 29, 2025, 12:20 AM UTC· resolved Oct 29, 2025, 12:36 AM UTC

1. investigating Oct 29, 2025, 12:20 AM UTC

   Our trans-Atlantic link from London to New York has dropped offline. Traffic is diverted via Ashburn to Frankfurt.

2. resolved Oct 29, 2025, 12:36 AM UTC

   This detour has already cleared and was a momentary disruption. TeraSwitch will engage our transport vendor for a better understanding of this incident.

Read the full incident report →

Started Oct 01, 2025, 09:52 PM UTC· resolved Oct 01, 2025, 11:30 PM UTC

1. investigating Oct 01, 2025, 09:52 PM UTC

   Teraswitch is investigating rising error rates and reports that Console is not working. During this time, management of services and ordering services may be unavailable. Customer services/servers and network operations are unaffected.

2. resolved Oct 01, 2025, 11:30 PM UTC

   All operations of Console services have been restored. Users should not see any issues managing their accounts and services.

Read the full incident report →

Started Sep 30, 2025, 07:01 PM UTC· resolved Oct 01, 2025, 02:13 PM UTC

1. investigating Sep 30, 2025, 07:01 PM UTC

   Teraswitch is currently investigating and monitoring rising temperatures at SLC1. We are awaiting comment from our facility vendor about what may be the cause. At this time, we are aware of no impacts to services or customer systems.

2. resolved Oct 01, 2025, 02:13 PM UTC

   This temperature issue was quickly resolved yesterday and was identified to be caused by maintenance work being completed on a nearby HVAC unit. The temperature increased due to lower airflow. At this time, all work is complete and the HVAC maintenance is completed. Our facility vendor is working to increase airflow so that in the future, specific units aren't required to supply the proper airflow.

Read the full incident report →

Started Sep 26, 2025, 12:00 AM UTC· resolved Jan 26, 2026, 01:57 PM UTC

1. monitoring Sep 29, 2025, 07:18 PM UTC

   Due to a sub-sea cable fault, our transport provider estimates an 10/17/2025 restoration to capacity between SGP1/2 to our backbone via Tokyo.

2. monitoring Dec 23, 2025, 04:01 PM UTC

   We received an update from our vendor that this will be repaired prior to Jan 31st. We have also purchased another diverse path to be installed in Jan 2026 to prevent the increased latency in the future.

3. resolved Jan 26, 2026, 01:57 PM UTC

   An alternative undersea cable has been established to restore this route. SGPTYO connectivity is now via backbone.

Read the full incident report →

Started Sep 24, 2025, 07:58 PM UTC· resolved Sep 24, 2025, 08:10 PM UTC

1. investigating Sep 24, 2025, 07:58 PM UTC

   Due to a probable fiber cut, EWR1/2 IAD1 has been detoured through alternative paths.

2. resolved Sep 24, 2025, 08:10 PM UTC

   The path has been restored and operations are normal between NJ and Ashburn. We are awaiting the provider's response about the cause of the disturbance.

Read the full incident report →

Started Sep 04, 2025, 09:43 PM UTC· resolved Sep 05, 2025, 12:59 AM UTC

1. identified Sep 04, 2025, 09:43 PM UTC

   Due to a fiber cut, our backbone has diverted Amsterdam Frankfurt traffic via London. There is no impact to operations other than higher latency between cities.

2. resolved Sep 05, 2025, 12:59 AM UTC

   The fiber cut has been restored and routing is now normalized.

Read the full incident report →

Started Sep 01, 2025, 11:18 PM UTC· resolved Sep 02, 2025, 03:26 PM UTC

1. monitoring Sep 01, 2025, 11:18 PM UTC

   Due to a network transport provider's routine maintenance having an issue, Teraswitch has elected to reduce risk and divert traffic from our backbone crossing through Frankfurt unnecessarily. At this time Frankfurt traffic will be routing locally and the global backbone will only use Frankfurt if required.

2. resolved Sep 02, 2025, 03:26 PM UTC

   Our network transport provider was able to roll back their changes with no impact. At a later date they will replace the malfunctioning hardware. Teraswitch has normalized operations in Frankfurt.

Read the full incident report →

Started Sep 01, 2025, 05:15 AM UTC· resolved Oct 17, 2025, 06:38 PM UTC

1. identified Sep 01, 2025, 05:15 AM UTC

   Teraswitch was alerted to a loss to our path between Seattle and Tokyo. We have directed traffic to external paths. We are working with our providers to resolve this. ULL traffic between Frankfurt and Tokyo may have seen ~5 minutes of degraded connectivity.

2. identified Sep 03, 2025, 05:50 PM UTC

   Our sub-sea cable vendor has confirmed that repairs should be completed around 9/19/2025. At this time, Tokyo backbone traffic remains diverted.

3. resolved Oct 17, 2025, 06:38 PM UTC

   Undersea cable operations have been restored, and traffic is stable and normalized along this path. Tokyo to Seattle is restored.

Read the full incident report →

Started Aug 26, 2025, 02:22 PM UTC· resolved Aug 26, 2025, 06:30 PM UTC

1. identified Aug 26, 2025, 02:22 PM UTC

   Teraswitch is rerouting traffic due to high latency seen with multiple internet providers in Singapore.

2. resolved Aug 26, 2025, 06:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 25, 2025, 10:13 PM UTC· resolved Sep 02, 2025, 03:26 PM UTC

1. investigating Aug 25, 2025, 10:13 PM UTC

   Teraswitch is investigating network instability issues affecting multiple sites. We will provide an update as more information becomes available.

2. identified Aug 25, 2025, 10:35 PM UTC

   The issue appears to be localized to a network transport vendor inside Frankfurt. Pass-through traffic would have been affected between cities. We believe that FRA1 and FRA2 were most impacted by this loss. Other providers in the city were affected so we are trying to understand the full scope of the issue. At this time network traffic is returning to normal.

3. resolved Sep 02, 2025, 03:26 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 19, 2025, 11:00 PM UTC· resolved Aug 20, 2025, 08:21 AM UTC

1. identified Aug 20, 2025, 12:44 AM UTC

   Due to a fiber cut on the Amsterdam path, Dublin (DUB1/2) traffic is diverted through London.

2. resolved Aug 20, 2025, 08:21 AM UTC

   The fiber has been restored.

Read the full incident report →

Started Aug 19, 2025, 08:45 AM UTC· resolved Aug 19, 2025, 06:03 PM UTC

1. monitoring Aug 19, 2025, 04:41 PM UTC

   Due to a fiber cut, all Amsterdam locations (AMS1/2/3) are diverted through London to get to Frankfurt (FRA1/2). There is no loss to connectivity other than higher latency via the detour.

2. resolved Aug 19, 2025, 06:03 PM UTC

   The fiber fault has been cleared and traffic is normalized.

Read the full incident report →

Started Aug 09, 2025, 02:03 AM UTC· resolved Aug 11, 2025, 02:28 AM UTC

1. investigating Aug 09, 2025, 02:03 AM UTC

   DAL1 IAD1 has lost connectivity. Traffic is diverted to alternative paths. Teraswitch is currently investigating and engaging our wave partners.

2. resolved Aug 11, 2025, 02:28 AM UTC

   All traffic is normalized.

Read the full incident report →

Started Aug 08, 2025, 02:12 PM UTC· resolved Aug 11, 2025, 02:27 AM UTC

1. identified Aug 08, 2025, 02:12 PM UTC

   SEA1 CHI1 has lost connectivity due to flapping wave service; traffic is diverted to alternative paths. Teraswitch is currently investigating and engaging our wave service providers.

2. resolved Aug 11, 2025, 02:27 AM UTC

   All traffic is normalized.

Read the full incident report →

Started Jul 25, 2025, 02:54 PM UTC· resolved Aug 11, 2025, 02:28 AM UTC

1. identified Jul 25, 2025, 02:54 PM UTC

   Teraswitch has identified issues relating to a loss of backbone capacity on the LON-EWR path. This loss of capacity was due to an emergency maintenance by the cable provider, causing various traffic shifts both on and off our network. Teraswitch engineering is rerouting traffic to mitigate congestion along alternate routes; customers in EWR2 and other sites may see degraded network performance until this issue is resolved. If you have any questions or concerns, please reach out to Teraswitch Support at [email protected]

2. monitoring Jul 31, 2025, 09:24 PM UTC

   Our undersea cable provider has restored access early to the cable system. At this time we do not see any issues with network traffic - but it may be possible we will have to divert traffic again.

3. monitoring Aug 06, 2025, 01:55 PM UTC

   The LON1 EWR2 path remains down while the cable is repaired.

4. monitoring Aug 06, 2025, 08:00 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Aug 11, 2025, 02:28 AM UTC

   This work is complete on the undersea cables and traffic flows are normalized.

Read the full incident report →

Started Jul 01, 2025, 01:56 PM UTC· resolved Jul 01, 2025, 09:04 PM UTC

1. investigating Jul 01, 2025, 01:56 PM UTC

   Teraswitch is monitoring an active cooling issue at our AMS1 site. Ambient temperatures are currently elevated and rising, and the facility has acknowledged an issue. We will continue to monitor the situation and provide updates as they become available.

2. identified Jul 01, 2025, 03:18 PM UTC

   Our facility provider for AMS1 has confirmed that the data center is having issues keeping up with internal cooling due to abnormally high outside air temperatures in Amsterdam, NL. We recommend that customers prepare for thermal shutdowns if temperatures continue to rise. At this time, we have not detected any loss of availability.

3. monitoring Jul 01, 2025, 05:12 PM UTC

   Ambient temperatures are now falling within AMS1. We believe that the risk of impact is over, but will continue to monitor the situation and provide updates until temperatures reach nominal levels.

4. resolved Jul 01, 2025, 09:04 PM UTC

   Ambient temperatures at AMS1 are now nominal once again per Teraswitch monitoring. Our facility provider has confirmed a solution was implemented and the cooling issue is now resolved. We will review the facility RCA once it is published. Teraswitch's network and services remained available throughout the incident. We apologize for any inconvenience caused by this event. If you have any questions, feel free reach out to Teraswitch Support at [email protected].

Read the full incident report →

Started May 17, 2025, 02:46 PM UTC· resolved May 17, 2025, 08:32 PM UTC

1. investigating May 17, 2025, 02:46 PM UTC

   Teraswitch is investigating a power loss event affecting at least one compute rack inside SGP1.

2. identified May 17, 2025, 03:05 PM UTC

   Teraswitch has isolated the issue to loss of power on a single compute rack in SGP1, causing downtime for bare metal customers in that rack. Customers in other racks at SGP1 are unaffected at this time. Teraswitch has engaged the facility to identify the cause and provide immediate restoration. We will provide an update once service has been restored.

3. monitoring May 17, 2025, 04:52 PM UTC

   The power issue affecting a single compute rack in SGP1 has been resolved - all affected customer systems should be back online at this time. Teraswitch staff are monitoring for stability but we do not expect further impacts.

4. resolved May 17, 2025, 08:32 PM UTC

   This incident has been resolved. A root cause analysis will be posted shortly.

5. postmortem May 19, 2025, 06:36 PM UTC

   On 5/17/2025, the facility provider for Teraswitch SGP1 \(Equinix SG3\) began scheduled maintenance on the B-side power bus for select data halls. This maintenance was expected to be non-service impacting to Teraswitch, as all our customer equipment and critical infrastructure are powered by redundant PSUs connected to diverse rack PDUs & facility power buses. At 10:00am UTC, the Equinix scheduled maintenance window began. At 01:38pm UTC, Teraswitch monitoring alerted to degraded power redundancy on SGP1 systems, which was expected given the scheduled maintenance. At 02:25pm UTC, Teraswitch monitoring provided the earliest indication of an issue affecting a single SGP1 compute rack. By 02:30pm UTC, additional alerts clearly indicated that this rack and associated addressing had gone offline. At 02:31pm UTC, Teraswitch engineering declared a likely power loss in the rack and contingency operations began. A Teraswitch technician located in Singapore was promptly dispatched to Equinix SG3, and a P1 smart hands ticket with Equinix was opened shortly thereafter at 02:56pm. At 03:28pm UTC, Equinix smart hands reported that the Bank 2 breaker on one of the PDUs in the affected rack had tripped, and all devices connected to it had consequently lost power. This suggested that an individual bank on the A-side PDU had been overloaded while the B-side power bus was offline for maintenance. At 03:46pm UTC, the Teraswitch technician arrived onsite and commenced work after clearing security. * After further investigation, it was confirmed that equipment in the rack had not been properly distributed across the PDU power banks. This had not caused an issue up until now since load is split across A and B-side PDUs under normal operating conditions - however, due to the facility B-side maintenance the rack load was transferred exclusively to the A-side PDU, and Bank 2 tripped due to the improper distribution. This caused a loss of power to much of the equipment in this rack. * Some customer equipment was connected to Bank 1 and did **not** lose power. However, Teraswitch network equipment was connected to Bank 2, causing a loss of network connectivity to all customer equipment in the rack regardless of power status. The Teraswitch technician redistributed power connections across the PDU banks on both A and B-side PDUs to avoid an overload of any one bank, and reset the Bank 2 breaker. At 04:25pm UTC, Teraswitch monitoring alerted to the final recovery of rack network equipment following power redistribution. Customer services recovered around the same time, within a few minutes before or after. ‌ Teraswitch has taken several measures in response to this incident: * In the affected SGP1 rack, power cabling was redistributed properly across the PDU banks, so there should be no future impact to that rack in any subsequent power events. * Teraswitch engineering evaluated PDU models in use globally that divide power capacity into banks, and added additional monitoring to notify of similar cases where an individual bank could be improperly loaded and its breaker could trip in a future power event. * As a result of the increased monitoring, Teraswitch staff identified two other racks in our global network that are susceptible to bank overload in their current configuration. Internal work tickets have been created to redistribute the power cabling in these racks to avoid any impact in a future power event. ‌ We apologize for any inconvenience caused by this incident and thank you for your continued support as a Teraswitch customer. If you have any questions or concerns, feel free to reach out to Teraswitch Support at [[email protected]](mailto:[email protected]).

Read the full incident report →

Started Apr 16, 2025, 04:58 PM UTC· resolved Apr 18, 2025, 06:58 PM UTC

1. monitoring Apr 16, 2025, 04:58 PM UTC

   Teraswitch has been monitoring multiple backbone connectivity losses in the United States. Connections between Pittsburgh and New Jersey were lost due to storms in NJ, and multiple US West fiber connections affecting the Los Angeles region were lost due to reported vandalism. At this time, the Teraswitch network has routed around the issues. Customers may see traffic detours and elevated latency. Some congestion may occur with certain paths between regions that are most closely impacted by the fiber disruptions.

2. monitoring Apr 17, 2025, 03:12 AM UTC

   All US Backbone connections have been restored. Networking is normalized and lowest latency paths are operational.

3. resolved Apr 18, 2025, 06:58 PM UTC

   This incident has been resolved.

Read the full incident report →