EX Networks experienced a major incident on February 3, 2024 affecting Volta (formerly Verne Global London), lasting 3h 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 03, 2024, 11:15 AM UTC
We are aware of power issues affecting Volta data centre and we are currently investigating.
- investigating Feb 03, 2024, 11:22 AM UTC
It appears that we have lost both power feeds on the 3rd floor and one power feed on the 4th floor. The data centre is investigating and we have dispatched an engineer to site.
- investigating Feb 03, 2024, 11:36 AM UTC
This outage coincides with a non service affecting scheduled maintenance taking place to upgrade the Engine Generator Paralleling System. This should not have impacted services. We are awaiting an official statement from Volta although our team are in constant communication with the on-site team.
- investigating Feb 03, 2024, 12:05 PM UTC
Our engineer has arrived on site in preparation for work required once power is restored. We do not have an ETA for power restoration at this time.
- investigating Feb 03, 2024, 12:21 PM UTC
The power feeds affected are A1 and B1. C1 is still operational. Work is underway to restore power but we do not have an ETA at this time.
- investigating Feb 03, 2024, 12:46 PM UTC
EXN continue to communicate with Volta/Verne and have received the following official updates: ***Update 12:45*** Verne Global Scheduled Maintenance Activity 240119-53593 During the maintenance activities we have experienced unexpected circuit breaker trips. Some customer load has been affected by the circuit breaker trips. Our onsite team is actively working to restore power to any customers that have been affected. We will continue to update to this email thread. We apologise for any inconveniences caused by these activities.
- investigating Feb 03, 2024, 01:29 PM UTC
Update from Verne: ***Update 13:30*** Verne Global Scheduled Maintenance Activity 240119-53593 We have identified the source of a control failure that was holding open circuit breakers feeding critical load on two of our systems. Customers that are fed from both of these systems will have been impacted by the maintenance event. We expect restoration of critical power within the next 90 minutes. We will then work to ensure that all cabinets that have been impacted are back online. Verne Global London DC Operations EXN is seeing power returning to the 3rd and 4th floor racks. Clients who require engineer resources to perform physical work in their Volta racks should raise a support ticket now in the usual way.
- monitoring Feb 03, 2024, 01:49 PM UTC
We are still waiting for official communications from Volta/Verne confirming the power restoration. EXN have observed power has returned to racks and we are monitoring. Clients who require engineer resources to perform physical work in their Volta racks should raise a support ticket now in the usual way.
- monitoring Feb 03, 2024, 02:08 PM UTC
Update from Verne: ***Update 14:00*** Verne Global Scheduled Maintenance Activity 240119-53593 Please be advised that as of 13:45 GMT we have restored power to the critical load systems that were impacted by the maintenance activity. We are working to ensure that all customer load is restored.
- resolved Feb 03, 2024, 02:59 PM UTC
We deem this incident to be resolved. We apologise for the inconvenience caused today. An RFO will be generated and distributed in due course. - please raise a ticket to ensure you receive a copy. If you are still experiencing issues or require on-site resources as a result of this outage, please raise a support ticket in the usual way.