RCA - SQL Database and App Service - West Europe

Update timeline

1. postmortem Feb 16, 2022, 03:31 PM UTC

   Summary of Impact: Between 07:31 UTC and 15:31 UTC on 16 Feb 2022, a subset of customers using SQL Database instances in West Europe may have experienced database connectivity errors in this region including when attempting to create new connections. Retries may have been successful. Additionally, customers utilizing Azure App Service in West Europe may have experienced issues while performing service management operations such as site create, delete, and move resources on App Service (Web, Mobile and API Apps) applications. Autoscaling and loading site metrics may have also been impacted. Root Cause: Due to a memory hardware failure in a network router, a control plane/data plane synchronization process in that router failed during a regular automated maintenance operation. This router was one of 8 redundant routers in that tier of the network, and the failure of the synchronization process led to the router dropping packets to a subset of the IP addresses it handled. The result was the failure of up to 12% of network flows to endpoints below the router. In particular, up to 12% of new connections to a subset of Virtual IP addresses (VIPs) would have failed. Retries would have likely succeeded and connections worked properly once established. As a downstream effect of the availability impact to some SQL services, App Service resources in the region with a dependency on SQL may have experienced issues. The mitigation took longer than expected as the impacted device was going through automated maintenance, during which alerts were suppressed. Alerts were triggered as expected by the automated alerting system once the device was brought back into rotation. This network device was being upgraded to a new firmware version which has, among other capabilities, the ability to automatically recover the control plane/data plane synchronization process in case of failure. This update helps prevent such failure scenarios in the future. Mitigation: • Engineers isolated the impacted network switch to mitigate the incident. • Engineers verified that no other network switches were impacted due to the same issue. Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): • Expedite the upgrade of network switches to the firmware that contains auto recovery of the control plane/data plane synchronization process. • Enhancements to automated maintenance and alerting platforms to improve alerting for devices undergoing maintenance. Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey