EX Networks incident
Global Switch 2 (East) - Loss of "B" feed power
EX Networks experienced a minor incident on May 3, 2022 affecting Global Switch London East (formerly Global Switch 2), lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 03, 2022, 07:16 AM UTC
At approximately 08:00 BST today EXN lost power to the "B" feed in Global Switch 2 (East) - 10th Floor Suite. The issue is being raised with the data centre. Further updates to follow.
- investigating May 03, 2022, 08:28 AM UTC
EXN have dispatched an engineer to site. ETA 20 minutes. Further updates to follow.
- investigating May 03, 2022, 08:43 AM UTC
EXN's engineer is on site and is moving single fed client devices on the "B" feed to the "A" feed. We're expecting an update from GS within 1 hour.
- investigating May 03, 2022, 09:01 AM UTC
All single fed client devices have now been moved to the "A" feed. If you are still experiencing issues, please contact our support team via the usual channels.
- investigating May 03, 2022, 10:56 AM UTC
Global Switch continue to investigate the issue with the "B" feed. Unfortunately we have no further updates at this time.
- investigating May 03, 2022, 12:39 PM UTC
We have recieved an update from Global Switch: This morning at approximately 8:10am Global Switch experienced an Emergency Mode on Power C. This is quite common and happens whenever the Powerstation notices a small fluctuation on the Mains supply. The Power Station switched to generators by design until the supply is healthy and then revert back to mains. Whilst the Station was running on Generators it appears LONE experienced a fault on the outboard supply which feeds Levels 3,8,9 and 10 which resulted in the Powerstation shutting down and going into Full Bypass. The ‘B’ supply is currently offline with the A supply active.
- investigating May 03, 2022, 06:13 PM UTC
We have recieved an update from Global Switch: Our specialist OEM contractors have confirmed that they need to complete additional intrusive testing and investigations. These works will be carried out using our planned, controlled process and procedures to protect your current supplies and services and the safety of all involved in this work. The switchboard that we are investigating is constructed in three sections and we need to isolate each section one at a time to carry out these investigations and confirm there are no issue with each section before it can be re-energised. This work is underway currently – specialist test equipment has been delivered to site to facilitate these works. We are expecting this work to be completed this evening. At present the "C" PowerStation is in direct by-pass and on utility supply and after the works are completed we will notify you before returning the system back to normal operation."
- investigating May 03, 2022, 07:27 PM UTC
We have recieved an update from Global Switch: Our specialist OEM contractors have commenced isolations to section one to carry out further investigations "C" PowerStation remains in direct by-pass and on utility supply and after the works are completed we will notify you before returning the system back to normal operation.
- investigating May 03, 2022, 09:04 PM UTC
We have recieved an update from Global Switch: Our specialist OEM contractors have now concluded testing to section one, now replacing covers and protection before moving on.
- investigating May 04, 2022, 02:37 AM UTC
We have recieved an update from Global Switch: We have now restored the B supply to the floors, this will remain as an open ring whilst our specialist OEM contractor continue to investigate the remaining sections of the switchboard
- investigating May 04, 2022, 06:34 AM UTC
At this stage Global Switch have advised PowerStation "C" remains in direct by-pass, with both A & B supply available. Switching will now commence to restore the DRUPS, whilst testing continues on the remaining breakers
- monitoring May 04, 2022, 07:02 AM UTC
We have recieved an update from Global Switch: "C" PowerStation is now online supplied via DRUPS, with both A & B supply on CLOSED rings"
- resolved May 04, 2022, 09:00 AM UTC
Global switch have completed all works and closed their incident. EXN can confirm all rack PDUs have been energised. We will be sharing the RFO/RCA once recieved by Global Switch to all affected clients.