Exascale incident

Loss of High Capacity Link from Wolverhampton (WLV) to Telehouse North (LHR)

Major Resolved View vendor source →

Exascale experienced a major incident on September 14, 2023 affecting LHR0 - Telehouse North, London, UK and Telford and 1 more component, lasting 7h 39m. The incident has been resolved; the full update timeline is below.

Started
Sep 14, 2023, 11:59 PM UTC
Resolved
Sep 15, 2023, 07:38 AM UTC
Duration
7h 39m
Detected by Pingoru
Sep 14, 2023, 11:59 PM UTC

Affected components

LHR0 - Telehouse North, London, UKTelfordWLV0 - Exascale, Wolverhampton, UKWest Bromwich

Update timeline

  1. identified Sep 14, 2023, 11:59 PM UTC

    The fibre provider between Wolverhampton and Telehouse North have identified a fibre break between two data centres in London. They have dispached engineers and we await an update. We expereniced issues rerouting the traffic in our Telford network when the fibre break taken place, this has now been resolved and service is restored. However service remains at risk until the fibre break in London is resolved.

  2. monitoring Sep 15, 2023, 01:47 AM UTC

    Zayo have notified Exascale that the fibre break has been resolved and line side ILAs have been enabled. As a result we have seen interfaces in both Wolverhampton and London recover. Traffic across our MPLS has now rerouted to the shortest path. We will continue to monitor and close this incident once we’ve observed 12 hours of stability.

  3. monitoring Sep 15, 2023, 01:48 AM UTC

    We are continuing to monitor for any further issues.

  4. resolved Sep 15, 2023, 07:38 AM UTC

    We’ve observed stability for a number of hours. We’re now closing this incident.