Probado experienced a minor incident on September 9, 2024 affecting VoIP (Voice over IP) and Data Services (Hosting), lasting 41d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 09, 2024, 10:21 AM UTC
We're aware of users encountering connectivity issues to the Manchester data and are actively investigating
- investigating Sep 09, 2024, 11:42 AM UTC
We are continuing to investigate the issue but are seeing services come back online. As the route cause has not yet been fully determined if you reconnect at this stage it is still with risk. Please ensure to save your work regularly. Further updates to follow.
- monitoring Sep 09, 2024, 12:13 PM UTC
Due to a "double fault" incident occurring at our datacentre partner (one power related and another routing based) Our Manchester services were knocked offline (full details to follow in post mortem) Services have now been restored and we are actively monitoring connections. While our partner continues to work on this issue we're advising customers that services are on line but considered 'at risk' until we get a clear notice from partner services. We apologise for the inconvenience caused and will be requesting a full RCA for the issue.
- monitoring Sep 12, 2024, 08:16 AM UTC
Maintenance is still being carried out by our datacentre partner in relation to the power loss events at the start of the week. We're continuing to actively monitor the situation and liaising to ensure client services are not affected (or where further disruption is expected prior notice is given) We have been assured that risk is minimal to services. But out of an abundance of caution, while a maintenance window is present, we are actively stating to clients that access to services should be considered as 'at risk' rather than declaring 'optimal' operation.
- resolved Oct 21, 2024, 08:10 AM UTC
We've had the all clear from our DC Partners to say maintenance on both power feeds are complete. We now consider this resolved
- postmortem Nov 08, 2024, 01:32 PM UTC
Suppliers RFO Posted Below: Reason for Outage \(RFO\) Report Incident: Temporary Loss of Power During UPS Maintenance Date of Incident: 09/09/2024 Prepared by: James Davenport Date of Report: 09/09/2024 # Incident Overview: During scheduled UPS maintenance on 09/09/2024, two temporary power outages occurred when transitioning the system to static bypass mode. The shared static bypass unit, serving all three UPS systems on the B stream, failed during both attempts. After additional checks and troubleshooting, the system was successfully transitioned to bypass, allowing safe work to proceed on replacing the failed units. Power has been stable since the final transition, and there are no long-term impacts on customer operations. # Incident Details: * Date & Time of First Incident: 09/09/2024, 11:15am * Date & Time of Second Incident: 09/09/2024, 13:25am * Date & Time of Successful Transition to Bypass: 09/09/202 * Duration of Power Loss for Each Incident: 50ms first incident, 15 seconds second # Cause of the Outage: The outages were caused by the failure of a shared static bypass unit during the transition to bypass mode. The static bypass unit, which is part of the aging infrastructure shared by all three UPS systems, failed twice during the switching process. After further checks and troubleshooting, the system was successfully transitioned to bypass, allowing the failed units to be safely bypassed for replacement. # Sequence of Events: 1. **First Incident:** The first power outage occurred during the scheduled maintenance when transitioning the system to static bypass mode. The shared static bypass unit failed, leading to a temporary loss of power to customer equipment. Power was restored quickly by reconfiguring the system. 2. **Second Incident:** After power was restored and additional checks were carried out, a second attempt to transition to bypass mode resulted in another failure. Power was once again quickly restored after bypassing the failed unit. 3. **Successful Transition:** After further troubleshooting and verification, the system was successfully transitioned to bypass mode without further incident. This allowed safe access to the faulty bypass unit for maintenance and replacement. # Contributing Factors: * Aging Equipment: The shared static bypass unit is over 20 years old and showed signs of significant wear. The failure in both instances was related to the deteriorating condition of this equipment and the reason why we embarked on a Critical infrastructure replacement scheme at Reynolds House. * Legacy System Challenges: Despite multiple pre-inspections and checks, the legacy infrastructure posed inherent risks during operational transitions. # Resolution: -Immediate Action Taken: Power was restored promptly after each failure. After further troubleshooting and testing, the system was safely transitioned to bypass mode, isolating the failed static bypass unit for maintenance. * Next Steps for System Repair: The failed static bypass unit will now be safely worked on and replaced as part of the ongoing infrastructure upgrades. # Corrective Actions: 1. Bypassing the Failed Units: The failed static bypass unit has been bypassed, allowing for safe work to proceed on its replacement. 2. Expedited Replacement of the Static Bypass Unit: The aging static bypass unit has been removed and will not be replaced. # Preventative Measures: 1. Complete Replacement of Legacy Systems: The static bypass unit and associated aging infrastructure has been removed, eliminating future risks during transitions and load transfers. # Conclusion: The failure of the shared static bypass unit caused two temporary outages during maintenance. After extensive checks and troubleshooting, the system was successfully transitioned to bypass, allowing the faulty units to be isolated and worked on safely. The replacement of the aging static bypass unit is now proceeding, with additional safeguards in place to ensure continued stability. # Next Steps: 1. Complete the replacement of the remaining aging UPS components. 2. Continue enhanced monitoring until the new system is fully operational. 3. Conduct a detailed review of other critical components and prioritize replacements for any remaining legacy systems. 4. Work with customers to make sure each rack has a fully redundant A and B power path.