RCA - Azure Resource Manager - Issues with management and resource operations

Update timeline

1. postmortem Jan 14, 2022, 08:00 PM UTC

   Summary of Impact: Between 09:00 UTC on 13 Jan 2022 and 20:00 UTC on 14 Jan 2022, a subset of customers using Azure Resource Manager (ARM) to deploy, modify, or remove Azure resources experienced delays, timeouts, and failures which were visible for long running operations executed on the platform. Impact was most severe for a period of 5 hours starting at 15:30 UTC on Jan 13 and another period of 8 hours starting at 00:00 UTC on Jan 14, and in regions including but not limited to West US, West US 2, South Central US, North Europe, West Europe, East Asia and Southeast Asia. Impact to customers will have been broad, as numerous Azure services rely on service management operations orchestrated by the ARM platform. Most customers will have experienced delays and timeouts, but many customers will have seen deployment or resource management failures. Root Cause: A code modification which started rolling out on 6 Jan 2022 exposed a latent defect in the infrastructure used to process long running operations (informally, "jobs"). The code modification resulted in an exception for a tiny fraction of job executions, each one of them disabled a small part of the job execution infrastructure. Over the course of hours, the job executions shifted entirely away from the regions that had received the new code to their backup paired regions. For a period of 16 hours, there was no customer impact as the backup paired regions executed the jobs as intended. The impact spread to backup paired regions as the new code was deployed, resulting in job queue up, latency delays, and timeouts. In some cases, the jobs executed with such prolonged delays that they were unable to succeed, and customers will have seen failures in these cases. As a result of the way that the job execution infrastructure was implemented, the compounding failures were not visible in our telemetry - leading to engineer’s mis-identifying the cause initially and attempting mitigations which did not improve the underlying health of the service. The consequence of this was a second period of impact starting at 00:00 UTC on 14 Jan 2022 and extending for approximately 8 hours. Mitigation: Identifying the source of the problems in this case took time, as some parts of the job infrastructure remained healthy and processing jobs, while other key parts were being disabled. At the time we were unable to clearly identify the newly released code as correlating with the impact we were seeing. When the nature of the problem became clear we immediately started to roll back to a previous build. This change rolled out progressively completed at 20:00 UTC on 14 Jan 2022. Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): • Reviewing and improving our monitoring and alerting strategy for our job execution infrastructure to improve our ability to detect problems like this one before they become customer-impacting. • Fixing the underlying problem which allows a single rare exception to disable parts of the job execution infrastructure. • Providing better visibility for operators when a paired region has assumed responsibility for job execution, in order to indicate a reduced-redundancy state and signal the need to pause or roll back a deployment. Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey