Is TeraSwitch down?

Teraswitch is currently tracking an outage on a cable between London, Dublin, and New York. Traffic has been detoured through Frankfurt and Washington DC for NY to London, and through Amsterdam for Dublin to London.

All links have been restored and are stable. Traffic is fully normalized.

Teraswitch has been tracking AWS Tokyo traffic taking long paths into TYO1/2/3. Local peering is disabled on the AWS side, and AWS transit traffic is selecting Singapore as a preferred hop over Tokyo. We are working on temporary solutions while AWS resolves the issue(s).

AWS has restored normal operation in the TYO market.

Due to a fiber cut between New York and Chicago, traffic has been diverted through Pittsburgh and other locations. Some traffic may be experiencing higher latency between PIT and NY/EWR due to the activation of high-latency but higher-capacity links.

The connection between NY and Chicago has been repaired and has remained stable for multiple hours.

(This status is split off from a previous issue that was originally believed to have been related.) Customers at LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3 experienced loss of reachability to Internet destinations, as well as to internal Teraswitch backbone destinations between affected sites. Other North American sites were not affected. MIA1 (Miami, FL) was intentionally removed from the backbone as part of our response. MIA1 remains reachable and in service, but is currently operating without full backbone connectivity, traffic to and from other Teraswitch sites is routed over the public Internet rather than our private backbone. Customers relying on MIA1 for private backbone or inter-site connectivity should expect changed latency and path characteristics until MIA1 is reintegrated. Cause: Teraswitch uses a default route (0.0.0.0/0) internally to signal that an edge router is able to forward traffic to the Internet. Each site normally prefers the default originated by its own local edge routers, with route attributes distinguishing a local origination from a remote one. A default route originated at MIA1 was propagated with its metric and communities stripped and an AS-path containing only our own ASN. At this time, our understanding is that this route never existed within MIA1's routers as a valid route. A route reflector at our AMS2 site propagated this altered route into our EU and APAC markets. Receiving edge routers interpreted it as locally originated and preferred it over their own valid local default. Those routers then advertised the route to the downstream data center core, which rejected it as invalid. With no acceptable default present, the affected site fabrics stopped forwarding traffic to their own edge routers, resulting in the loss of reachability observed. Resolution: Engineers identified the malformed route within 10 minutes of onset and removed MIA1 from the backbone to halt further propagation. Affected sites reconverged on their local default routes and service was restored at 04:16:15 UTC. Next steps: The underlying defect, whether in the MIA1 edge routers or in the route reflector, has not yet been identified. We are working to reproduce the condition and have engaged our vendor. In the interim we are implementing policy to enforce minimal required attributes on internally originated default routes so that a malformed advertisement cannot be preferred over a valid local one. MIA1 will remain off the backbone until the defect is understood and mitigations are verified; we will post an update when it is reintegrated. A full RFO will follow.

Working with our network hardware vendor, we have deployed a global change across all compute sites to improve the reliability of the connection between our Internet/backbone edge routers and our data center core networks. This change addresses the blackhole condition that caused the overnight outage. With it in place, a malformed route entering the fabric will no longer prevent traffic forwarding - sites will continue to forward via their local edge routers rather than losing reachability. We considered the previous behavior a critical flaw and have prioritized this fix accordingly. All sites, including the unaffected data centers, had this configuration change completed successfully earlier today. We continue to investigate the underlying condition that caused the default route from MIA1 to be advertised/readvertised with incorrect attributes, and will publish a full RFO once that analysis is complete. Please note: a separate, unrelated incident affecting our global traffic routing systems remains open. That work involves a distinct vendor defect and route scaling improvements, and is not connected to the outage described here.

_\(A more detailed/illustrated RCA in PDF format is available on request\)_ # Internet Outage Event at Multiple Sites **Incident date:** August 12, 2026  ·  **Regions affected:** EU, APAC  ·  **Full recovery:** 33 minutes ## Executive summary In the early morning of August 12, 2026 \(UTC\), customers at thirteen of Teraswitch's 31 sites lost reachability to Internet destinations and to other Teraswitch sites over the private backbone: twelve across the EU and APAC regions, and MIA1 in North America, which was intentionally and temporarily removed from the backbone as a containment measure. The cause was a stale default route \(`0.0.0.0/0`\) originated by an MIA1 edge router: a static route left over from the site's turn-up, redistributed into BGP. The static is deliberately less preferred than any transit-learned default and had sat inactive for months, until a routine transit provider maintenance withdrew the local external default shortly before the first alarm and activated it. As originated, the route carried an empty AS path containing no external networks, a metric of 0, and no communities: the ordinary signature of a redistributed static route. A digit transposition in a route-map on MIA1's sessions to the global route reflectors, `20326:301:x` written where `20326:130:x` was intended, then placed the route into the community lists that control regional export, and the global reflectors added the no-export community to the copies sent toward the EU and APAC regions. The route reflector hierarchy propagated it to our EU and APAC markets carrying that flag. Because the attributes that normally distinguish a remote default from a local one were never attached, receiving edge routers at the affected sites evaluated the route as locally originated and preferred it over their own valid local default. No-export is a control flag that forbids a route from being advertised across a BGP boundary into another ASN. The connection from the edge routers to the data center fabric is exactly such a boundary. When the affected edge routers selected the stale route as their best default, no-export prevented them from advertising any default to their spines. With no default route present, the affected fabrics stopped forwarding Internet-bound traffic, isolating them from their own healthy edge routers. The first alarm fired at `03:49:52 UTC`. Engineers identified the stale route within 10 minutes; the remainder of the outage window was spent executing containment, removing MIA1 from the backbone and powering off the AMS2 route reflector, and waiting for routes to recalculate. Reconvergence began at `04:16:15 UTC`, with service returning progressively as sites withdrew the stale default and reconverged on their local default routes; routes fully recalculated and traffic returned to normal 33 minutes after the first alarm. A global configuration change deployed to all compute sites later the same day ensures that a stale or invalid route entering the fabric can no longer prevent traffic forwarding. **Sites affected:** 13 of 31 · **Time to identify:** Edge Traffic Two observations made the true failure harder to see. First, traffic was still visible flowing from the edge layer to the spine layer at affected sites. The key detail, recognized later, was that this was L2 transport and private VRF traffic. Only the `INTERNET` vrf had failed, and the presence of healthy traffic on the same links delayed a closer look at the fabric's Internet routing specifically. Second, management access looked completely normal, for reasons described below. ### Management network behavior during the outage Teraswitch maintains a completely separate management network at each site, fronted by a high availability firewall pair. Under normal conditions these firewalls use the Teraswitch backbone, but they also connect to our serial console servers, which carry a dedicated and isolated Internet connection that is entirely separate from Teraswitch operation. When the data center networks became unusable, the firewalls failed over to this out of band connectivity automatically. This design worked exactly as intended: our team had unfettered access to every device throughout the outage, as though everything were normal. For the same reason, it was slightly misleading. Because access felt normal, it appeared that nothing major was wrong with the equipment or the fabrics, when in fact the management plane had instantly routed around the very failure under investigation. Good for resolution and access, but misleading for diagnosis. ### The moment of identification Despite the emergency response not recording the broken state of the network, the responding engineer captured the routing state on the AMS3 edge at the moment they located the issue: ``` er111a.ams3.teraswitch.com#show ip route 0.0.0.0/0 VRF: default Source Codes: C - connected, S - static, K - kernel, O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2, N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type2, B - Other BGP Routes, B I - iBGP, B E - eBGP, R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2, O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary, NG - Nexthop Group Static Route, V - VXLAN Control Service, M - Martian, DH - DHCP client installed default route, DP - Dynamic Policy Route, L - VRF Leaked, G - gRIBI, RC - Route Cache Route, CL - CBF Leaked Route Gateway of last resort: B I 0.0.0.0/0 [200/0] via 100.96.49.1/32, IS-IS SR tunnel index 103 via TI-LFA tunnel index 40, label 800961 via 64.130.60.122, Ethernet25/1, label imp-null(3) backup via 64.130.60.81, Ethernet26/1, label 974060 er111a.ams3.teraswitch.com#show bgp ipv4 unicast BGP routing table information for VRF default Router identifier 100.124.63.2, local AS number 20326 Route status codes: s - suppressed contributor, * - valid, > - active, E - ECMP head, e - ECMP S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast % - Pending best path selection Origin codes: i - IGP, e - EGP, ? - incomplete RPKI Origin Validation codes: V - valid, I - invalid, U - unknown AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop Network Next Hop Metric AIGP LocPref Weight Path * > 0.0.0.0/0 100.96.49.1 0 - 100 0 ? Or-ID: 100.96.63.1 C-LST: 20.32.6.119 20.32.6.1 ``` This is the stale default as received at an affected edge: origin incomplete \(the marker of a redistributed route\), an AS path containing no external networks \(making it appear locally originated\), a next hop of `100.96.49.1` in MIA1 addressing, and reflection metadata, originator ID `100.96.63.1` with the AMS2 cluster list, showing the path the route took. The edge itself could resolve that next hop over an IS-IS SR tunnel across the backbone, so from the edge's perspective the route was usable. What made it unusable as the fabric signal was the no-export community it carried: once selected as best, it could not be advertised across the BGP boundary to the spines. Next-hop-self is applied on the edge to spine sessions, so next hop reachability played no part in the fabric failure. ## Root cause analysis ### Confirmed observations * The default route was received by EU and APAC edge routers without the standard communities that identify a legitimate remote default, and with an empty AS path containing no external networks: as originated, the bare signature of a locally redistributed static route. As received, it also carried the transposed large-community values described below and the no-export community, established by configuration analysis and lab replication of the full issue. * The route carried the originator ID of MIA1 edge router er100a.mia1 \(`100.96.63.1`\) and a cluster list of `20.32.6.119 20.32.6.1`: the AMS2 regional reflector and a global reflector, most recent reflector listed first. This matches MIA1's reflector topology at the time. MIA1's edge routers were connected directly to the global reflectors, a valid but intentionally temporary arrangement used because the edge routers were installed before the site's own reflector units, so no MIA1-local or North American regional entries would be expected on this path. * At the time of the event, er100b.mia1 was observed properly learning its external default route, and both MIA1 edge routers were receiving defaults from our other regional North American route reflectors. The stale static default sat inactive on er100a.mia1 by design, activating and entering BGP only when er100a.mia1's own external default was withdrawn during the provider maintenance that began around 03:30 UTC. * Receiving edge routers preferred the route because the attributes that distinguish local origination from remote origination were absent. * The route carried the no-export community, applied by the global reflectors' regional export policies through the transposed community values. Once an edge router selected this route as its best default, no-export forbade advertising any default across the BGP boundary to the spines, which under the then-current design removed the fabric's only Internet default. * Next-hop-self is configured on the edge to spine sessions and operated as designed. Next hop reachability played no role in the fabric failure. * No Teraswitch automation run or configuration change triggered the event; the trigger was an external provider maintenance. The vulnerability it exposed, however, was our own: a standing configuration error, the transposed community values, deployed well before the event. ### The default route left on MIA1 The origin of the route was identified in er100a.mia1's configuration: a static default route left over from site turn-up, still present and still redistributed into BGP. ``` "static_routes": [ { "destination": "0.0.0.0/0", "nexthop": "100.105.0.2", "vrf": "default" } ] ``` The static was configured when the site operated on a single transit provider and should have been removed when the second transit was activated. Its next hop, `100.105.0.2`, is a Dallas edge router directly connected to er100a.mia1. Because our edge routers prefer transit-learned defaults over static ones, it sat inactive in the background for months, doing nothing, until the transit provider maintenance withdrew the external default and activated it. A same-day audit of every edge router found no other static default routes, and edge router configuration templates have been updated to disallow static default routes entirely. IS-IS has also been blocked from redistributing default and static routes. ### How the no-export was applied Routes from MIA1's edge routers reached the global reflectors over direct sessions, the temporary arrangement described above. An inbound policy on those sessions stamps arriving routes with regional classification values: ``` route-map RM-NA-INBOUND permit 10 set large-community 20326:301:30 20326:301:20 20326:130:10 additive ``` The intended values belong to the `20326:130:x` family, which marks normal routes meant to be sent to all regions: ``` bgp large-community-list standard REGION-NA-EXPORT permit 20326:130:10 bgp large-community-list standard REGION-EU-EXPORT permit 20326:130:20 bgp large-community-list standard REGION-APAC-EXPORT permit 20326:130:30 ``` The first two values written contain a digit transposition: `20326:301:30` and `20326:301:20` where `20326:130:30` and `20326:130:20` were intended. The third, `20326:130:10`, was written correctly. The `20326:301:x` family is reserved for the opposite purpose, regional export control: ``` bgp large-community-list standard REGION-NA-NOEXPORT permit 20326:301:10 bgp large-community-list standard REGION-EU-NOEXPORT permit 20326:301:20 bgp large-community-list standard REGION-APAC-NOEXPORT permit 20326:301:30 ``` The global reflectors' export policies toward each region add the well-known `no-export` community to any route carrying that region's value. The goal of this mechanism is containment, not removal: a route marked this way remains present and usable within AS20326 in those regions, but is never exported to transit or peering there. This is a different case from a separate blocking mechanism in the same policies, which prevents a route from being present in a region at all. The edge to spine sessions cross a BGP boundary of the same kind as transit and peering, which is how a flag aimed at external announcements was also able to suppress the fabric signal. The EU policy is shown; the APAC policy is identical in structure: ``` route-map RM-EU-OUTBOUND permit 40 match large-community REGION-EU-NOEXPORT set community no-export additive ``` Every route MIA1 sent over the direct sessions was therefore marked for no-export toward EU and APAC. The NA value, `20326:301:10`, was never written by the transposition, so copies distributed within North America carried no flag. This policy is defined for direct edge to reflector sessions and was attached only at MIA1, the only site operating without its own local reflectors. Validation testing at the time did not cover the edge to global reflector operating mode combined with static default routes existing on the edge router; our testing expansion and configuration sanity checking now do. No capture displaying the route's communities survived from the event window. The chain described here was confirmed by replicating the event in a lab environment using pre-outage configuration state. ### The trigger and remaining open items The origin and the trigger are now confirmed. The leftover static default on er100a.mia1 activated when the provider maintenance described earlier withdrew the local external default. The route entered BGP, was stamped with the transposed community values on MIA1's direct sessions to the global reflectors, and received the no-export community toward EU and APAC as a result. The transposition was not caught in validation testing because testing did not cover the temporary operating mode of an edge router connected directly to the global reflectors; that testing gap has since been closed. Had MIA1 not been removed from the backbone, the event would eventually have resolved itself: when the provider maintenance completed and the external default returned, it would have displaced the static and the route would have been withdrawn. The interaction between the two MIA1 edge routers is resolved: er100b.mia1 never accepted er100a.mia1's static default route, because the BGP policy between the two edge routers intentionally does not share static routes. This is why MIA1 itself initially stayed online: er100b.mia1 continued feeding both MIA1 spines a working default carrying no no-export flag. One item remains under review: why the EU regional reflector at AMS2 emitted the route toward its clients while the second EU regional reflector, hosted in Frankfurt, did not. The AMS2 reflector's running state, which would have answered this directly, was lost when it was powered off during containment. This document will be revised as it closes. ### Contributing factors * **Readiness signaled with external routes rather than a locally originated default.** Edge routers did not generate an artificial, clean default route of their own to signal readiness to the data center networks. Instead they re-advertised default routes learned from external transit providers, prepared for that role by route-maps. The readiness signal therefore carried external attributes and inherited external failure modes, and a stale or invalid route could still win best path and stand in for a valid signal. * **Fail-closed fabric behavior.** With the edges' best default carrying no-export and therefore barred from the fabric sessions, the fabric was left with no default at all and withdrew Internet forwarding entirely. There was nothing to fall back on: routing does not retain a departed route, and no lower-preference default existed to activate in its place. We considered this a critical design flaw and prioritized its correction. The two available solutions were a lower-preference default that activates when the expected one is lost, or force-advertising the default from the edge routers; we chose force-advertisement, deployed as part of the August 12 hardening. * **Misleading healthy signals during diagnosis.** L2 transport and private VRF traffic continued to flow from edge to spine, masking that only the `INTERNET` vrf had failed, and management access remained fully functional after failing over to out of band connectivity, suggesting equipment and fabrics were healthy. The failover did raise alarms, but within the wave of critical alarms accompanying the onset, nothing specific to the management network rerouting stood out. Together with a geographic pattern resembling a DDoS event and a recent unrelated IS-IS instability, these signals directed early effort away from fabric Internet routing. * **Evidence capture was manual.** Service restoration rightly took priority during the response, and debug capture required manual effort. While our responding engineer guessed correctly that the issue was with MIA1's default export, we do not have a capture of the no-export community on the route from the moments during the outage. We supplemented this by replicating the issue in our lab. * **Incorrect generalization from a prior event.** A November 2024 default route incident at FRA2 was caused by our own maintenance tooling in combination with a simultaneous route flap at an outside provider. Its remediation led to the assumption that default route failures would originate from within our own change processes, and the design was never evaluated against a stale or invalid update arriving via the backbone itself. See the following section. * **A digit transposition, unprotected by testing.** The inbound policy on the direct edge to global reflector sessions was written with `20326:301:x` values where `20326:130:x` was intended, placing every MIA1 route into the regional no-export control lists. Validation testing did not exercise the temporary operating mode of an edge router connected directly to the global reflectors, so the error was never caught before the event. ## Prior related event and design assumptions In the interest of a complete and honest accounting: this was not the first time a default route problem isolated a Teraswitch data center fabric. Teraswitch was aware of a related failure mode from an incident at FRA2 on November 27, 2024, beginning at 09:35 UTC. The conclusions we drew from that event shaped a design assumption that the August 12, 2026 outage proved incorrect. In both events, a default route carrying no-export won selection on a site's edge routers, and the fabric behind them lost its signal. In both, a routine provider event touching the local default exposed the latent flag. ### The November 27, 2024 FRA2 event The root cause was a combination of unfortunate provider timing and a logical error in the BGP tooling used to divert traffic away from links about to undergo maintenance: 1. The maintenance tooling set a `no-export` flag \(BGP community\) on all routes learned from the AS20326 backbone. Inadvertently, this included the default route \(`0.0.0.0/0`\) that is exported toward the data center network to signal that an edge router is ready to pass Internet traffic. 2. Initially there was no impact: FRA2's edge routers had selected a default route from a local FRA2 Internet transit provider as that signal. 3. Possibly affected by the same fiber vendor maintenance, one or more local FRA2 transit providers regenerated their default route, causing the FRA2 edge routers to elect a new best default: one generated at a nearby location \(FRA1\) and learned over the backbone, still carrying the `no-export` flag. 4. The FRA2 edge routers therefore stopped exporting a default route toward the FRA2 data center network \(routers within a different ASN than AS20326\). The data center concluded that no edge router was available to process Internet traffic, causing a total FRA2 outage until the flag was removed. ### Remediation completed after the 2024 event * The application of the `no-export` flag to default routes was removed entirely. It has no traffic steering effect, since Internet destinations almost always have a more specific route within the Teraswitch network. * A safety was added to the edge routers' route-maps that strips all BGP communities from routes intended to be sent down to the spines, so a control flag can never again suppress the advertisement of the default route unintentionally. As the 2026 event showed, this protection is structurally limited: the well-known no-export community blocks advertisement before any outbound route-map runs, so a route arriving already flagged is never offered to the route-map at all. * A route-map on the edge routers was configured to never set the `no-export` or `no-advertise` communities on a default route sent up to the reflectors or to other edge routers. ### The assumption we drew, and why it was wrong The 2024 event stemmed from our own tooling, triggered when an outside provider happened to flap its routes at the same time; the fix was within our own change process. From this, Teraswitch concluded that a falsely applied no-export would come from our own edge routers under maintenance, and that tooling and process corrections at the edges were sufficient protection. We never accounted for, or considered, the no-export community being added anywhere else: by a route reflector, or by an external ASN. Nor did we consider a default route arriving as "local" but not actually being usable as the readiness signal for the fabric spines. Those assumptions were incorrect. We remained vulnerable to the same flag arriving from any origin other than the one we had specifically resolved. During the August 12, 2026 event the community was already present on the stale MIA1 route when it reached the affected edge routers. It was applied by our own global reflectors' direct-edge import policies. This bypassed all changes and improvements from the 2024 event. The global fabric hardening deployed on August 12, 2026 closes this design flaw: the edge routers now force-advertise a locally generated default whenever they are genuinely able to carry Internet traffic, so the fabric always holds a valid default while its edges are healthy, regardless of where the edges learn their own default route or what control communities that route carries. ## Resolution and recovery Engineers identified the stale route within 10 minutes of onset. MIA1 was removed from the backbone by shutting down the loopback interface used for its SR-MPLS/IS-IS connectivity, and the AMS2 route reflector was powered off, halting further propagation and speeding recalculation. At that moment the responding engineer did not yet know whether MIA1 itself was the source or the AMS2 reflector was malfunctioning; what was known was that the affected sites had learned the route through AMS2's reflection, so both were removed. The team also halted route exports from the global route reflectors in the same window. The only operational change from that halt was peering traffic taking local and regional paths rather than crossing regions; customer and internal routes were maintained throughout by our Continuity Reflectors, which handle only internal and customer routes. With the route's origin now confirmed as er100a.mia1, removing MIA1 from the backbone withdrew the stale default at its source and restored service; powering off the AMS2 reflector removed the EU distribution path and forced recalculation. Reconvergence began at `04:16:15 UTC`. The AMS2 reflector shutdown left a large backlog of route changes, so withdrawing and reprogramming the default route took time, and service returned progressively as each site reconverged on its local default routes. Full recovery, with routes recalculated and traffic patterns back to normal, took 33 minutes from the first alarm. MIA1 returned to the backbone through a maintenance shortly after the event. Later the same day, working with our network hardware vendor, we deployed a global configuration change across all compute sites, including unaffected data centers, to improve the reliability of the connection between our Internet/backbone edge routers and our data center core networks. With this change in place, a stale or invalid route entering the fabric will no longer prevent traffic forwarding: sites will continue to forward via their local edge routers rather than losing reachability. The rollout favored safety over speed. The configuration was updated on the edge routers feeding approximately 62 spine devices globally, one edge router at a time, completing before 15:00 UTC. At that point we did not yet know where the route had come from, so the global route reflectors were kept offline for the duration of the rollout to halt any propagation if it happened again, and the network ran on regional routes only. ### How the fix works The previous design had each edge router share the default route it had itself selected as the readiness signal toward the spines. The new design uses the BGP `default-originate` feature on the neighbor group from the edge routers toward the spines. The edge now generates the default route itself, so the route sent into `vrf INTERNET` always carries the local edge router as its next hop and can never inherit a next hop or attributes from a stale or invalid route elsewhere in the backbone. Advertisement of this default is controlled by our automation system rather than by a route policy. When maintenance or a traffic drain requires an edge router to stop attracting traffic, the automation deploys a flag on that edge router that stops the advertisement, and removes the flag to restore it. A route-map on the session shapes the attributes of the originated default but does not control whether it is originated. This preserves the original design goal, that an edge router does not attract traffic from the spines before it is ready, retaining the blackhole protection the shared-default design was built for while removing the dependency on a route that transits the backbone. ### Route reflector rebuild and software diversity As further hardening, all backbone route reflectors were wiped and fully rebuilt with the latest operating system and reflector software in the week following the event. We also introduced reflector software diversity: a portion of the reflectors now runs a completely different software package, so a defect in any single implementation can no longer affect every reflector at once. This also hedges against the class of defect that lives in an interaction between specific software and firmware versions rather than within any single component. ## Corrective actions 1. Deploy global fabric hardening: replace shared default route signaling with `default-originate` from the edge routers toward the spines, so a stale or invalid route entering the fabric cannot prevent traffic forwarding. \(**Complete**, Aug 12, 2026\) 2. Wipe and fully rebuild all backbone route reflectors with the latest operating system and reflector software, and introduce reflector software diversity by running a portion of the reflectors on a completely different software package. \(**Complete**, Aug 12, 2026 and following days\) 3. Reintegrate MIA1 into the backbone. \(**Complete**, Post-event maintenance\) 4. Correct the transposed community values and audit all route-maps for values colliding with control community families. \(**Complete**, Week of Aug 10, 2026\) 5. Increase logging and automatic debug state capture across backbone devices, including BMP for BGP monitoring, so forensic evidence is preserved without slowing service restoration. \(**In progress**, Ongoing\) 6. Deploy per-VRF default route monitoring: spine and leaf devices raise an alarm when a VRF that previously carried an active default route no longer has one. \(**In progress**, Ongoing\) 7. Move customer DoubleZero interconnects to within the data center fabric, so tunnel sessions no longer depend on paths outside it. \(**Planning**, TBD\) 8. Expand validation testing to cover alternative operating modes, including edge routers connected directly to the global reflectors. \(**Complete**, Post-event\) 9. Audit every edge router for leftover static default routes \(none besides MIA1 were found\), and update edge router configuration templates to disallow static default routes entirely. \(**Complete**, Aug 12, 2026\) 10. Block IS-IS from redistributing default routes and static routes. \(**Complete**, Post-event\) 11. Review and rebuild the BGP community scheme network-wide, an ongoing project that predates this event. \(**In progress**, Ongoing\) 12. Reclassify management network failover to out of band connectivity as a higher priority alarm than most other alerts, so it stands out during an alarm storm. \(**Planning**, TBD\) ## Lessons learned * Signal our own gear with a route we originate ourselves. When a default route is used to tell our downstream devices that an edge is ready, that route should be generated locally for that purpose, not borrowed from an external source that carries its own attributes and failure modes. * The fate of routing between two devices should be decided between those two devices. A forwarding decision on a local link should not be inherited from a different site, a transit provider, or any other distant route source; the new design enforces this on the edge to spine boundary. * Signals that look healthy can hide a broken plane. Traffic on shared links and fully functional management access both masked a failure confined to the `INTERNET` vrf. A failover whose alarms are buried in a larger alarm storm, as happened when the management firewalls moved to out of band connectivity, can easily go unnoticed during the response. * Service restoration is priority #1 in these outages; engineers are only worried about restoring access. Evidence preservation must therefore be automatic: increased logging and automatic debug capture will preserve forensic state without slowing recovery. * Fixing a root cause is not the same as fixing a failure mode. The 2024 remediation addressed the specific trigger \(a tooling error\) rather than the general class of failure \(any untrustworthy default route reaching the fabric\). The broader vulnerability remained until August 12, 2026. ## Closing statement This event was our mistake, even though triggering it required several small and unrelated conditions to align: a leftover route, a transposed digit, and a routine provider maintenance. We take pride in our choices and designs, and this event has driven us to reevaluate multiple other systems to verify their own integrity. The design carried an assumption about what could break: that any default route would be fully "cleaned" by its edge routers before it could affect the fabric behind them. This event proved otherwise: the no-export flag was honored rather than cleaned, withholding the fabric's default while every edge router operated normally. This assumption held for six years, but the 2024 FRA2 event was not properly considered beyond the specific triggers of that day. We are also not blind to what you are thinking after reading this: losing a default route inside our own fabric looks stupid. It is one of the most basic functions in networking, and it was not protected as well as it needed to be. If another provider did this, we would chuckle at it and be glad we had not made such a mistake. We learn from other providers' issues and successes all the time; perhaps someone will learn from ours. At the same time, the surrounding design performed the way it was built to. Management access was retained to every device for the full duration, the offending route was identified within ten minutes, and affected sites recovered in roughly half an hour. This network absorbs failures every day, of equipment, routes, fiber optic spans, undersea links, and power, and customers rarely notice, because surviving those failures is where most of the design effort goes. This is the first event of this scale since the backbone took its current form in 2020, in a company operating since 2003. We also understand the size and the sensitivity of what runs on this network. We remain unbiased about our customer base and about what customers choose to host with us, but we know exactly what earned that base and what set us apart from our competition: people trust us not to have failures. Simultaneous data center failures, in particular, are not acceptable to us. Our change control and management systems have always been built to make that outcome impossible, and every data center is designed to keep running through the loss of every other data center, in what we call island mode. In this event, several data centers all happily accepted the same bad route, and that has changed our view of how issues can propagate. We would have preferred to find this failure mode in validation testing rather than in production, and that is the lasting change: the bar for validation and edge case testing has been greatly raised. The specific condition behind this event can no longer occur, the remaining hardening is being deployed in deliberately sequenced steps so that remediation never becomes its own event, and the network is stronger today than it was on August 11. **Brendan Mannella** Chief Executive Officer **Nick Zurku** Principal Network Architect ## Appendix ### A. Affected sites * **EU** \(LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2\): Internet and inter-site backbone reachability lost * **APAC** \(SGP1, SGP2, TYO1, TYO2, TYO3\): Internet and inter-site backbone reachability lost * **NA** \(MIA1\): Temporarily removed from backbone as containment; operated in island mode with Internet routing over local transit only, later reintegrated via maintenance * **NA** \(All other NA sites\): Not affected ### B. Terminology * **default route \(0.0.0.0/0\):** A catch-all route covering all Internet destinations. Teraswitch uses it internally to signal that an edge router is able to forward traffic to the Internet. * **vrf INTERNET:** The routing instance on the spine layer that carries Internet routes for the data center fabric. * **route reflector:** A router that redistributes BGP routes between sites so that every router does not need a direct session with every other router. * **IS-IS:** The interior routing protocol Teraswitch uses to distribute reachability information inside its own network. * **out of band \(OOB\):** Management connectivity that is fully independent of the production network, used to reach equipment even when the production network is impaired. * **next hop:** The address a router must be able to reach to actually use a route. If the next hop cannot be resolved, the route is invalid and cannot be installed for forwarding. * **BGP attributes:** Metadata carried with a route \(communities, metric, AS path\) used to identify its origin and control which route is preferred. * **no-export:** A standard BGP community instructing routers not to advertise the route across a BGP boundary into another ASN. * **edge / spine / leaf:** The three layers of a Teraswitch site: edge routers face the Internet and backbone, spines aggregate the site, and leaf switches connect customer servers.

Teraswitch is looking into general Internet Instability.

Teraswitch is currently working on mitigations related to this issue, although traffic issues have settled. At this time, it appears a bulk of the high-capacity US to EU cables had a fault around the same time. Our NY to London path, which had been cut and resolved earlier today, went down again, and our Ashburn to Frankfurt path suffered an outage around the same time. At this time, the network has stabilized and we are working to ensure any additional events are not as impactful to traffic.

Since the initial impact, Teraswitch’s Network Engineering team has identified the instability as a routing software failure triggered during a routine configuration change. We have removed the triggering condition and implemented safeguards to prevent recurrence. The network remains stable while we complete software upgrades and permanent remediation. Our current timeline of events is that the instability was triggered at approximately 19:09:21–27 UTC on July 29th 2026, and lasted for ~5 minutes. More updates and a complete RCA will be released in future updates.

Teraswitch is investigating reports and alerts of similar issues affecting Internet connectivity.

Teraswitch has completed remediations to remove the bug-triggering configurations from our routers, as well as upgrading our route reflectors and adding diverse vendors on routing software. These stabilizations have proven reliable since July 30th, when we have not seen any additonal invalid packets (IS-IS LSA advertisements) inside our backbone. We expect no further impacts from this issue.