Exascale incident

Loss of Hypervisor in Telford

Major Resolved View vendor source →

Exascale experienced a major incident on September 13, 2025 affecting Telford and West Bromwich and 1 more component, lasting 5h 37m. The incident has been resolved; the full update timeline is below.

Started
Sep 13, 2025, 04:42 PM UTC
Resolved
Sep 13, 2025, 10:19 PM UTC
Duration
5h 37m
Detected by Pingoru
Sep 13, 2025, 04:42 PM UTC

Affected components

TelfordWest BromwichTEL0 - Exascale, Telford, UK

Update timeline

  1. identified Sep 13, 2025, 04:42 PM UTC

    The DNS servers in Telford have failed due to being hosted on the Hypervisor cluster which has failed. We recommend setting your DNS servers to 8.8.8.8 or 4.4.4.4 to restore service while we resolve the issue.

  2. identified Sep 13, 2025, 05:01 PM UTC

    We have introduced other DNS servers into the network, service should be starting to restore.

  3. identified Sep 13, 2025, 05:44 PM UTC

    We have restored the majority of services, however we are currently rebuilding the hypervisor to restore Radius which is affecting national broadband circuits delivered over Openreach.

  4. identified Sep 13, 2025, 05:44 PM UTC

    We are continuing to work on a fix for this issue.

  5. monitoring Sep 13, 2025, 08:28 PM UTC

    We have recovered all critical services and non-critical services will start to recover shortly.

  6. resolved Sep 13, 2025, 10:19 PM UTC

    All systems are now back to normal.