Exascale incident

Loss of high capacity link between Telford and Wolverhampton (MBORC)

Minor Resolved View vendor source →

Exascale experienced a minor incident on April 28, 2024 affecting Telford and West Bromwich and 1 more component, lasting 16h 43m. The incident has been resolved; the full update timeline is below.

Started
Apr 28, 2024, 10:25 PM UTC
Resolved
Apr 29, 2024, 03:08 PM UTC
Duration
16h 43m
Detected by Pingoru
Apr 28, 2024, 10:25 PM UTC

Affected components

TelfordWest BromwichTEL0 - Exascale, Telford, UK

Update timeline

  1. investigating Apr 28, 2024, 10:25 PM UTC

    We are currently investigating this issue.

  2. investigating Apr 28, 2024, 11:18 PM UTC

    We're still awaiting an update from the Dark Fibre carrier, however it is understood that there is Fibre cable damage in Wolverhampton affecting multiple operators.

  3. identified Apr 29, 2024, 01:30 AM UTC

    Vandals have cut a number of fibre cables on the entry into our Wolverhampton point of presence. Third party contractors have been engaged to pull new cables urgently.

  4. identified Apr 29, 2024, 03:16 AM UTC

    Engineers will be onsite at 5:30AM to pull in 70m of new cable and carry out splicing. We’re hoping to restore service before 9am

  5. identified Apr 29, 2024, 06:47 AM UTC

    Splicers are now on-site to connect the strap cable to the existing cables either end. Resolution ETA 9am

  6. identified Apr 29, 2024, 08:08 AM UTC

    Further damage has been found and engineers are working through each fibre optic joint to repair ETA 10am

  7. identified Apr 29, 2024, 09:30 AM UTC

    Engineers are working on the final splice closure we expect to start to see services recovering within the next 60 minutes.

  8. identified Apr 29, 2024, 10:08 AM UTC

    The Engineers have fitted 3 of the 4 fibres in the new splice closure at the front end of the estate and they should begin splicing around 11:00. They have advised within the next hour they will be able to confirm splicing of the first 3 fibres is complete. A second Splicer from COMEX has been requested to attend site

  9. identified Apr 29, 2024, 11:07 AM UTC

    Splicing teams are prioritising fibres and advise services should start to recover very soon.

  10. identified Apr 29, 2024, 12:46 PM UTC

    Splicing teams are now moving onto the final splicing cloure and services should start to recover very soon.

  11. identified Apr 29, 2024, 01:27 PM UTC

    We've been advised there are two remaining splices and service should be restored and we will receive the all clear

  12. monitoring Apr 29, 2024, 03:08 PM UTC

    Service has been restored. We will continue to monitor the situation as traffic now starts flowing.

  13. resolved Apr 29, 2024, 03:08 PM UTC

    This incident has been resolved.