EX Networks experienced a notice incident on February 5, 2024, lasting 1d 11h. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Feb 05, 2024, 03:54 AM UTC
At approximately 03:28 at Volta this evening, racks powered be the B riser went down before recovering at 03:43. The A and C risers remained opperational during this.
- identified Feb 05, 2024, 06:18 AM UTC
At 06:07 the B riser went down again, before recovering at 06:17.
- identified Feb 05, 2024, 07:17 AM UTC
Update from Volta/Verne: We are aware of UPS B system losing supply twice this morning. This is still related to the issue we experienced over the weekend. A procedure is in place to reset the system. The fault is due to equipment on the system which was unforeseeably damaged during planned switching works. Replacement equipment is on order, and we are working as hard as we can to put a fix in place to prevent the system losing power until the replacement equipment arrives.
- identified Feb 05, 2024, 03:08 PM UTC
Update from Volta/Verne: Data Centre remains stable with all 3 UPS systems online. We are currently working on a fix that can be put in place until such time as the replacements for the faulty equipment can be received and installed. Fix is expected to be started after 5pm GMT today and will be implemented 1 switch board at a time. More detail will be released once the works are fully planned. UPS B switch board continues to be manually monitored at all times.
- identified Feb 05, 2024, 04:17 PM UTC
Update from Volta/Verne: We have planned works to install temporary power supplies to replace the faulty equipment identified in the LV switch panels. We will commence a simulation of the procedure at 6pm GMT and will send an update out once complete to outline the schedule of proposed works. The simulation will not affect customer supply.
- identified Feb 05, 2024, 07:44 PM UTC
Update from Volta/Verne: The simulation test has been successfully concluded, and all aspects proceeded without issues. We are planning to proceed to UPS B at 20:00 GMT. UPS B and associated switch panel will remain operational. Should there be any urgent concerns, please address them prior to this scheduled time.
- identified Feb 05, 2024, 09:00 PM UTC
Update from Volta/Verne: The UPS B switchboard works have been successfully completed, and we are now progressing to UPS A.
- monitoring Feb 05, 2024, 09:47 PM UTC
Update from Volta/Verne: All scheduled works have been completed successfully. There will be no additional works until further communication is provided.
- resolved Feb 06, 2024, 03:18 PM UTC
The current status is that the data centre is in normal operations and all equipment will function as expected in a mains failure event. Further work is required for a permanent solution which we will communicate in the usual way.