Exascale incident

Loss of uplink from WLV to MAN0

Minor Resolved View vendor source →

Exascale experienced a minor incident on January 25, 2023 affecting WLV0 - Exascale, Wolverhampton, UK and MCR0 - Datum, Manchester, UK, lasting 20h 38m. The incident has been resolved; the full update timeline is below.

Started
Jan 25, 2023, 10:57 AM UTC
Resolved
Jan 26, 2023, 07:35 AM UTC
Duration
20h 38m
Detected by Pingoru
Jan 25, 2023, 10:57 AM UTC

Affected components

WLV0 - Exascale, Wolverhampton, UKMCR0 - Datum, Manchester, UK

Update timeline

  1. investigating Jan 25, 2023, 10:57 AM UTC

    We're currently observing the loss of a high capacity link from Wolverhampton to Manchester. All traffic is routing via other paths at present, albeit at higher latency.

  2. identified Jan 25, 2023, 07:43 PM UTC

    Zayo (the provider of the high capacity link) have advised a fibre cable carrying our traffic has been damaged in the Stoke on Trent area. Zayo are currently trying to find enough slack to resplice.

  3. identified Jan 25, 2023, 09:00 PM UTC

    The Zayo fibre team are working hard to fix this fibre break, One side has enough fibre to pull through, they're now checking the other side.

  4. identified Jan 25, 2023, 09:20 PM UTC

    The Zayo fibre team continue their efforts to restore service. Unfortunately there is not enough slack on the existing cable. As such their civil teams are assisting to excavate the area and a new cable will need to be installed and additional excavation work is required in order for replicing to begin. Due to conditions and location Zayo currently do not have an ETR but have been advised a tentative estimate of 2 hours before preparation works will be completed and splicing can begin.

  5. resolved Jan 26, 2023, 07:35 AM UTC

    Service has now been restored. Traffic has now moved to its shortest path.