Infrastructure Supplier Outage

Update timeline

1. resolved Aug 19, 2024, 09:05 PM UTC

   At 10.35 on 19/08/2024, our monitoring systems detected an outage, which we quickly identified as a network-related issue. Upon investigation, Intuto immediately contacted our infrastructure provider. They informed us that they were doing routine maintenance that was taking longer than expected and had caused the outage unexpectedly. At 1.30am our monitoring systems indicated that the outage had been resolved. We are awaiting full details from infrastructure provider and will update this item with full details when available.

2. postmortem Nov 21, 2024, 11:16 PM UTC

   ### **Postmortem: Infrastructure Supplier Outage** **Incident Date:** 19/08/2024, 10:00 PM **Duration:** 2 hours and 55 minutes **Summary:** On Monday 19th August 2024 at 22:00, our infrastructure provider began planned maintenance activity, to replace a faulty B feed power breaker in our a datacenter which was scheduled to take place between 10:00pm and 6:00am the following day. The expected impact of this change was a B feed power interruption, which posed a risk to services if energy consumption in the customer rack was exceeding provisioned requirements. At 23:19 the provider started received queries relating to cloud service impacts and began investigating. It was identified that several devices in the cloud services rack were offline. Further investigation on site at the datacenter concluded that following B power being turned off for the scheduled work, the load on the remaining PDU was unbalance and tripped a breaker, leaving the rack with only one of 1 powered circuits, dropping power feeds to all vSphere hosts and resulting in a total cloud services outage. The tripped breaker was switched back on and a partial rollback of the scheduled maintenance was completed so that B power could be restored to the rack. Engineers began powering hardware back on and performing health checks. Restoration activities were completed by 2:39am, with all monitored services reporting operational. Cloud engineers continued to work with customers for any VM recovery requirements. The planned breaker replacement was completed successfully by 6:00am **Root Cause:** Planned maintenance by our infrastructure supplier caused an unexpected outage. The following root cause was provided. > Load requirements in cloud services rack are too high for single power feed. Unbalanced PDU load in Cloud services rack dropped power feeds to vSphere hosts. **Impact:** The outage disrupted the availability of all services hosted on the affected infrastructure, making them inaccessible to end users for 2 hours and 55 minutes. Since the incident occurred overnight during a typically low-usage period, the impact on users was minimal. **Conclusion:** This incident highlights the importance of proactively identifying and mitigating potential risks during planned maintenance activities. While the maintenance itself was necessary, the outage could have been avoided with better preemptive measures to assess power load distribution and redundancy. We have been advised that the following actions have been taken by our provider, to minimise the risk of these issues recurring in the future. * Review of notification process for Datacentre works. A new process has been put in place to improve the capability to notify customers and stakeholders of planned maintenance and risks. * Review of Cloud service rack PDUs. Scheduled work will take place in future to shift infrastructure to less loaded racks. We sincerely apologize for any inconvenience this outage may have caused and are committed to delivering reliable services. The lessons learned from this incident will guide our continuous improvement efforts. We apologize for any inconvenience caused and remain committed to ensuring the stability and performance of our system. If you have any questions or concerns, please don’t hesitate to [reach out to our support team](mailto:[email protected]). **The Intuto Team**