I.T Communications Limited incident

London leased lines - Outage

Critical Resolved View vendor source →

I.T Communications Limited experienced a critical incident on June 5, 2019 affecting Customer Leased Lines, lasting 8h 6m. The incident has been resolved; the full update timeline is below.

Started
Jun 05, 2019, 11:43 AM UTC
Resolved
Jun 05, 2019, 07:50 PM UTC
Duration
8h 6m
Detected by Pingoru
Jun 05, 2019, 11:43 AM UTC

Affected components

Customer Leased Lines

Update timeline

  1. investigating Jun 05, 2019, 11:43 AM UTC

    We are seeing some issues in the Covent Garden exchange area affecting leased lines services and confirm that the issue is caused by Openreach fibre damage. We are working with Openreach engineers to fix this as soon as possible.

  2. identified Jun 05, 2019, 11:45 AM UTC

    While we are seeing some lines recover intermittently, we are still in the process of working with Openreach to ensure proper resolution.

  3. monitoring Jun 05, 2019, 11:55 AM UTC

    We are seeing circuits back online and stable. We are continuing to monitor whilst we await for the resolution from Openreach

  4. resolved Jun 05, 2019, 07:50 PM UTC

    This incident has been resolved.

  5. postmortem Jun 06, 2019, 03:00 PM UTC

    An Openreach engineer on site at the Covent Garden exchange caused disruption to fibre connections terminated there affecting a subset of customers leased lines. CAUSE OF INCIDENT Following alerts on our Network monitoring platform, engineers identified the affected circuits involved were connected to the Covent Garden exchange, where an Openreach engineer had been working. A site visit from our engineers and Openreach found the fault to be with the path taken of fibres within the fibre tray causing disruption when this was opened and closed. RESOLUTION DETAILS Openreach engineers met with our engineers at the Covent Garden exchange. Some fibres were identified as taking an incorrect path through the splicing tray which would have meant that their bend radius would have been exceeded when opening and closing the tray. These fibres were re-guided via their correct channels which should now restore reliable connectivity. FOLLOW-UP ACTIONS We will liaise with Openreach to ensure that all fibre connections follow the correct channels in future.