EX Networks incident
Fibre Down between Telehouse East and Virtus Enfield
EX Networks experienced a notice incident on November 10, 2020 affecting DC Connect and IP Transit Services and 1 more component, lasting 2d 10h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 10, 2020, 10:53 PM UTC
We have been alerted to another fibre break between Telehouse East (THE) and Virtus Enfield (ENF). The link is currently down and we are raising the fault with our fibre provider. All traffic has re-routed via an alternative link and services remain stable in ENF.
- investigating Nov 10, 2020, 11:17 PM UTC
At approximately 23:09 the fibre appears to have restored. We are costing out this link until we hear back from our fibre provider. At this stage fully redundancy has been restored however we consider this link "AT RISK".
- monitoring Nov 11, 2020, 12:58 AM UTC
Services remain stable while our fibre provider looks into the issue.
- monitoring Nov 11, 2020, 05:52 AM UTC
Our fibre provider has an engineer on site at THE to conduct further testing of this circuit. All services currently main stable however should intrusive testing be required, the circuit may be disconnected while OTDR measurements are taken.
- monitoring Nov 11, 2020, 09:04 AM UTC
The ENF-THE fibre was bought down at 07:28 for OTDR testing. The test results are currently being analysed. We are pushing our fibre provider for a quick resolution.
- monitoring Nov 11, 2020, 10:14 AM UTC
We have observered the circuit come back online. Traffic is still reouted away from the circuit however redundancy has been restored. We are awaiting a response from our fibre provider.
- monitoring Nov 11, 2020, 11:53 AM UTC
Fibre engineers have completed the OTDR tests from THE however these are inconclusive. The engineers are currently on their way to ENF to conduct further tests, so we expect this link to go down again while testing. Further updates to follow.
- monitoring Nov 11, 2020, 04:10 PM UTC
Our fibre provider has completed their testing and found an issue within the ODF at Enfield. They have given the all clear however we will continue to monitor this link while we await the full RCA from our vendor. We have stressed for this to be completed urgently and for them to conduct a full review of the link as the number of breaks is unaccetable. Further enquiries will be made after the RCA has been received. All services continue to operate normally.
- monitoring Nov 12, 2020, 10:00 AM UTC
Services have remained stable since our last update. We are contiuning to monitor this link.
- resolved Nov 13, 2020, 09:33 AM UTC
This incident has been resolved.