Exascale Outage History

Started Oct 06, 2026, 10:45 PM UTC· resolved Oct 07, 2026, 06:31 AM UTC

1. investigating Oct 06, 2026, 10:45 PM UTC

   We are observing an outage on our Vodafone Broadband Interconnect, affecting broadband services delivered through this interconnect. We are in the process of reporting the issue to Vodafone’s service desk. Further updates will follow.

2. investigating Oct 06, 2026, 11:06 PM UTC

   The incident has been raised with Vodafone’s service desk as a Priority 1 (P1) fault under reference FLT13710040 and escalated due to the repeated nature of the fault. We have confirmed that the fault is outside the Exascale network. Our LNS infrastructure is responding as expected, but the Vodafone interface remains hard down. Optical light levels remain steady, indicating a fault at the remote A end within Vodafone’s network site. Our next update will follow at approximately 1:00am BST on 7 October.

3. investigating Oct 07, 2026, 12:08 AM UTC

   We have not yet received an update from Vodafone’s service desk. We are contacting them now to request an urgent update and will share further information as soon as it becomes available.

4. investigating Oct 07, 2026, 12:16 AM UTC

   Vodafone has provided a new fault reference: NET13709772. The fault has been escalated due to its repeated nature. Vodafone has advised that a further update is expected within 10 minutes, at approximately 1:26am BST on 7 October. We will share further information as soon as it is received.

5. investigating Oct 07, 2026, 12:41 AM UTC

   Vodafone has advised that they are currently assigning an engineer to attend the MCR11 and MCR12 network sites. The engineer will check equipment, cabling, SFP optical modules and chassis line cards as part of the investigation. We will provide a further update as soon as more information is available.

6. investigating Oct 07, 2026, 02:36 AM UTC

   Vodafone reports that the affected circuit remains down. Field engineers are assigned to attend both ends of the circuit to check patch cabling, replace the optical transceivers (XFPs) and, if required, change the ports. Vodafone has advised that its next update is expected at 4:00am BST on 7 October.

7. investigating Oct 07, 2026, 04:42 AM UTC

   In its latest update, sent at 4:14am BST on 7 October, Vodafone advised that both field engineers were still travelling to the affected network sites and that the circuit remained down. The engineers are to confirm their arrival and begin their checks in coordination with Vodafone’s network engineers. Vodafone has not confirmed service restoration.

8. investigating Oct 07, 2026, 05:58 AM UTC

   In its latest update, sent at 5:51am BST on 7 October, Vodafone confirmed that an engineer had arrived at one of the affected network sites. A replacement engineer has been assigned to the other site, where Vodafone is arranging access. The circuit remained down, and Vodafone has not confirmed service restoration.

9. monitoring Oct 07, 2026, 06:27 AM UTC

   Service appears to have recovered. Our checks show that the Vodafone interface and BGP session are back up and have remained up for 37 minutes as of 7:25am BST on 7 October. The incident has been moved to monitoring while we await further updates from Vodafone.

10. resolved Oct 07, 2026, 06:31 AM UTC

    Vodafone has confirmed that its engineers replaced an SFP optical transceiver, clearing the alarms, and that services are operating within normal ranges. Following this confirmation and our earlier interface and BGP recovery checks, this incident is now resolved. We apologise for the disruption.

11. postmortem Oct 09, 2026, 03:03 PM UTC

    Reason for outage – supplier findings Vodafone confirmed on 9 October 2026 that the outage was caused by a faulty SFP optical transceiver at its MCR11 network site. The transceiver was replaced and service was restored. Vodafone also installed an optical attenuator as a precaution to prevent potential transceiver damage from high receive power. Vodafone has not confirmed that high receive power caused the failure. We will continue to monitor the interconnect and report any further issues promptly. We have requested a formal RFO report from Vodafone, including clarification on the repeat nature of the fault and measures to prevent a recurrence. The incident remains resolved.

Read the full incident report →

Started Oct 05, 2026, 01:59 AM UTC· resolved Oct 05, 2026, 11:58 AM UTC

1. investigating Oct 05, 2026, 01:59 AM UTC

   The interface, facing the Vodafone Wholesale Broadband network is hard down. We have logged the issue with Vodafone Service Desk.

2. identified Oct 05, 2026, 07:08 AM UTC

   Vodafone has identified the failed component within their network. A field engineer is currently on site reconfiguring the network to migrate our uplink to an alternative chassis at their MCR11 network site. This work is intended to bypass the affected equipment and restore connectivity while the underlying fault is addressed.

3. identified Oct 05, 2026, 08:03 AM UTC

   Vodafone have advised that their field engineer is expected to arrive on site at 9:30am today to move our uplink to alternative, operational equipment within their network. This work is intended to restore service, although restoration cannot be confirmed until the move and subsequent testing are complete. We will provide a further update as soon as more information is available.

4. identified Oct 05, 2026, 08:04 AM UTC

   Vodafone have advised that their field engineer is expected to arrive on site at 9:30am today to move our uplink to alternative, operational equipment within their network. This work is intended to restore service, although restoration cannot be confirmed until the move and subsequent testing are complete. We will provide a further update as soon as more information is available. No components were marked as affected by this incident.

5. identified Oct 05, 2026, 09:13 AM UTC

   Vodafone has confirmed that work is required at both MCRN11 and MCRN12. An engineer is currently arranged for MCRN11, and Vodafone has requested an additional visit to MCRN12 to complete the required port move. As the sites are close together, Vodafone has requested that the same engineer attend MCRN12 once the work at MCRN11 is complete. We’ll continue to provide updates as work progresses. Next update by 11:10 AM on 5 October.

6. identified Oct 05, 2026, 10:31 AM UTC

   The interface connecting Exascale to Vodafone’s network has recovered, and we are beginning to see customers’ routers reconnect. We are monitoring service recovery and stability. Full restoration has not yet been confirmed, and we will provide a further update once we have verified that all affected services have recovered. Thank you for your patience, and we apologise for the disruption.

7. monitoring Oct 05, 2026, 10:43 AM UTC

   We are monitoring service recovery and stability. Full restoration has not yet been confirmed, and we will provide a further update once we have verified that all affected services have recovered.

8. resolved Oct 05, 2026, 11:58 AM UTC

   This incident has been resolved. Vodafone has confirmed that the circuit has remained operational and stable, with traffic passing normally for over 60 minutes. Vodafone will carry out a further stability check overnight and arrange a planned replacement of the affected card at MCRN11. Thank you for your patience. We apologise for the disruption caused.

Read the full incident report →

Started Aug 22, 2026, 07:12 AM UTC· resolved Aug 22, 2026, 08:24 PM UTC

1. investigating Aug 22, 2026, 07:12 AM UTC

   We have reported the issue to Vodafone and we await further updates.

2. identified Aug 22, 2026, 07:54 AM UTC

   Vodafone are aware of a transmission fault in Manchester engineers are attending the site in the Metroplex Business Park

3. identified Aug 22, 2026, 08:28 AM UTC

   Vodafone have advised that there is a faulty SFP which has failed and requires intimidate replacement.

4. identified Aug 22, 2026, 10:13 AM UTC

   We have chased for a further update but one is not yet available.

5. identified Aug 22, 2026, 10:53 AM UTC

   Vodafone have advised parts are being sourced and engineering resource dispatched to replace faulty optical transmission equipment.

6. identified Aug 22, 2026, 01:06 PM UTC

   Vodafone engineer is now onsite attempting to resolve.

7. identified Aug 22, 2026, 01:22 PM UTC

   A second Vodafone engineer is attending the B-end of the circuit to carry out an End to End (E2E) test.

8. identified Aug 22, 2026, 02:07 PM UTC

   Vodafone have advised it is currently suspected to be one of the XFP’s. They should have the ETA of these engineers within the next 30 – 60 minutes – as soon as we have their ETA I’ll keep you informed.

9. identified Aug 22, 2026, 02:24 PM UTC

   The first engineer to arrive at one of Vodafone’s network nodes should be there for 16:30 and will begin investigations independently. The second network node is located at a data centre site, with an ETA between 16:30 and 17:00. Further updates will be provided if any of these times change, and will provide further updates once we receive any further feedback from Vodafone field engineers.

10. identified Aug 22, 2026, 04:46 PM UTC

    The first engineer has arrived and checked the equipment at the first network node, in which they have found no issue. They are en route to the other node at the data centre location. Further updates will be available within the next 60 minutes.

11. identified Aug 22, 2026, 05:44 PM UTC

    The engineer has attended the second network node, and has replaced the SFP on site. However, we are still seeing a Loss of Signal alarm. Vodafone transmission engineers are working on a slot change to assist in providing a resolution. Further updates will be provided once this work is completed.

12. identified Aug 22, 2026, 07:23 PM UTC

    The engineer has attended the second network node, and has replaced the SFP on site. However, we are still seeing a Loss of Signal alarm. Vodafone network engineers are working remotely with the Vodafone field engineer and are conducting loop tests to investigate further, 3rd line support engineers will be engaged if further assistance required.

13. monitoring Aug 22, 2026, 08:06 PM UTC

    We have seen the interface facing Vodafone recover, we’re also observing L2TP sessions reconnect. Traffic leaving the LNS is also building. We will continue to monitor.

14. resolved Aug 22, 2026, 08:24 PM UTC

    We’re confident after speaking with Vodafone Service Desk the interface is now stable and has recovered after all the XFPs and SFPs have been replaced in path. If your broadband service hasn’t recovered please email [email protected]

Read the full incident report →

Started Aug 16, 2026, 12:42 PM UTC· resolved Aug 16, 2026, 07:29 PM UTC

1. identified Aug 16, 2026, 12:42 PM UTC

   We have observed a core router reload in Telehouse North, while the majority of the network has continued to function any directly connected circuits to pe0.lhr0.uk.as61049.net will have been interrupted. We are raising a case with Cisco TAC.

2. monitoring Aug 16, 2026, 12:56 PM UTC

   We are starting to see BGP sessions restore on pe0.lhr0.uk.as61049.net

3. resolved Aug 16, 2026, 07:29 PM UTC

   We have seen stability for many hours. We are to schedule a future IOS XR upgrade to improve stability.

Read the full incident report →

Started Mar 18, 2026, 09:12 PM UTC· resolved Apr 13, 2026, 12:21 PM UTC

1. monitoring Mar 18, 2026, 09:12 PM UTC

   We have seen observed elevated optical loss and increased errors towards agg1, we have taken decisive action to replace the optical transceivers at both ends of the link to ensure service remains stable.

2. monitoring Apr 13, 2026, 12:20 PM UTC

   We are continuing to monitor for any further issues.

3. monitoring Apr 13, 2026, 12:21 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Apr 13, 2026, 12:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 26, 2026, 10:00 AM UTC· resolved Feb 26, 2026, 10:00 AM UTC

1. resolved Feb 26, 2026, 10:43 AM UTC

   An unexpected outage occurred in the data centre, impacting the network. This has since been resolved.

Read the full incident report →

Started Nov 18, 2025, 11:53 AM UTC· resolved Nov 18, 2025, 06:58 PM UTC

1. identified Nov 18, 2025, 11:53 AM UTC

   We are experiencing an outage with all websites and services protected by CloudFlare.

2. identified Nov 18, 2025, 11:56 AM UTC

   We're starting to see websites recover. We await further updates from CloudFlare.

3. identified Nov 18, 2025, 12:01 PM UTC

   Unfortunately we have seen services via CloudFlare revert to being offline. We await updates.

4. resolved Dec 23, 2025, 06:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 13, 2025, 04:42 PM UTC· resolved Sep 13, 2025, 10:19 PM UTC

1. identified Sep 13, 2025, 04:42 PM UTC

   The DNS servers in Telford have failed due to being hosted on the Hypervisor cluster which has failed. We recommend setting your DNS servers to 8.8.8.8 or 4.4.4.4 to restore service while we resolve the issue.

2. identified Sep 13, 2025, 05:01 PM UTC

   We have introduced other DNS servers into the network, service should be starting to restore.

3. identified Sep 13, 2025, 05:44 PM UTC

   We have restored the majority of services, however we are currently rebuilding the hypervisor to restore Radius which is affecting national broadband circuits delivered over Openreach.

4. identified Sep 13, 2025, 05:44 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Sep 13, 2025, 08:28 PM UTC

   We have recovered all critical services and non-critical services will start to recover shortly.

6. resolved Sep 13, 2025, 10:19 PM UTC

   All systems are now back to normal.

Read the full incident report →

Started Jul 12, 2025, 06:51 AM UTC· resolved Jul 12, 2025, 03:45 PM UTC

1. investigating Jul 12, 2025, 06:51 AM UTC

   We’re currently observing a fibre break towards our West Bromwich network. We have raised a fault with the supplier.

2. identified Jul 12, 2025, 07:47 AM UTC

   After further investigation we are seeing a line-card has failed in our Wolverhampton POP. This particular card is hosting all the local Openreach circuits facing businesses and also our West Bromwich network. We are attempting to reseat the card to remedy the issue.

3. identified Jul 12, 2025, 07:47 AM UTC

   We are continuing to work on a fix for this issue.

4. identified Jul 12, 2025, 08:21 AM UTC

   Reseating the linecard has not resolved the issue, we have placed an order for a replacement line card but this will not arrived till Tuesday of next week. We are currently working a temp solution to restore service in the mean time. Next update in 2 hours.

5. identified Jul 12, 2025, 08:37 AM UTC

   We have sourced temp hardware and an engineer will be installing circa 12 noon and re-routing the circuits facing the faulty line card.

6. identified Jul 12, 2025, 12:48 PM UTC

   Engineers are on-site implementing the work around.

7. identified Jul 12, 2025, 12:55 PM UTC

   The workaround has now been implemented and we are starting to see services recover.

8. monitoring Jul 12, 2025, 12:55 PM UTC

   We're now monitoring the situation.

9. resolved Jul 12, 2025, 03:45 PM UTC

   We have seen traffic passing for a number of hours and are happy the temp fix has resolved the issue and brought services back into service. A maintiance window will be required to carry out the permanent fix next week.

Read the full incident report →

Started Jun 11, 2025, 12:49 PM UTC· resolved Jun 12, 2025, 08:02 AM UTC

1. investigating Jun 11, 2025, 12:49 PM UTC

   We’re currently experiencing circa 40 Ethernet circuits not forwarding layer 2 traffic in our Telehouse North point of presence. We have reported the issue to Vodafone NOC and have rebooted Vodafone’s equipment as a precautionary measure. We await further updates.

2. monitoring Jun 11, 2025, 01:31 PM UTC

   We’ve started to observe circuits recover, however we still have no updates from Vodafone. We will provide further information when available.

3. monitoring Jun 11, 2025, 03:30 PM UTC

   Vodafone have advised that a line card at a Vodafone Redstream site experienced a failure (crashed). The faulty line card was removed and reseated, which successfully restored service.

4. resolved Jun 12, 2025, 08:02 AM UTC

   We've observed circuits being stable for over 12 hours, we're now closing. If you're still having issues please email [email protected]

Read the full incident report →

Started Mar 18, 2025, 10:36 PM UTC· resolved Apr 13, 2025, 09:22 PM UTC

1. identified Mar 18, 2025, 10:36 PM UTC

   AMS-IX have advised Exascale of an issue effecting the PE router of which Exascale is directly connected. AMS-IX advise traffic has been switched using photonics to a shadow PE router while investigations are on-going.

2. resolved Apr 13, 2025, 09:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 24, 2025, 11:43 AM UTC· resolved Jan 29, 2025, 01:07 AM UTC

1. monitoring Jan 24, 2025, 11:43 AM UTC

   We are actively monitoring our network and can confirm that Exascale is not experiencing any issues related to Storm Éowyn. Our network remains stable and fully operational. As an extra precaution, our Exascale Fleet is conducting regular checks on the network assets every few hours. There’s no need for concern—this is simply an additional step we’re taking to ensure continued stability during this time. Update from Openreach Openreach has informed us that their teams are extremely busy due to the ongoing disruption caused by Storm Éowyn. You can view their release here: https://openreach-comms.co.uk/cr/AQischCr0IoHGJzot7YH-IzPvLB6gigUY0TiFqorr8eGF11nfLq1-35Z6Rkuw_I We appreciate your cooperation and understanding. If you have any questions or need further assistance, please reach out to our support team at [email protected].

2. resolved Jan 29, 2025, 01:07 AM UTC

   The Exascale network didn’t incur any damage as part of this weather event.

Read the full incident report →

Started Jan 04, 2025, 09:19 PM UTC· resolved Jan 05, 2025, 08:33 AM UTC

1. investigating Jan 04, 2025, 09:19 PM UTC

   We are currently observing a number of circuits down which are provided by Vodafone in the West Midlands area. We're currently awaiting a sitrep from Vodafone

2. identified Jan 04, 2025, 11:06 PM UTC

   Vodafone Fixed Line NOC have advised there is a Fibre break in Birmginham Small Heath Area approx 15km from a Vodafone Red Stream site. The is no ETA at present as the break is still being investigated. Updates will be provided as received.

3. resolved Jan 05, 2025, 08:33 AM UTC

   We have seen Ethernet circuits and WDSL recover at around 1:05am this morning and has remained stable.

Read the full incident report →