Azure Data Factory V2 - West Europe - Mitigated

Update timeline

1. postmortem Jan 14, 2022, 08:00 PM UTC

   Summary of Impact: Between 10:10 UTC on 13 Jan 2022 and 20:00 UTC on 14 Jan 2022, a subset of customers may have experienced issues, timeouts, or failures for some service management operations for services leveraging Azure Resource Manager (ARM). This could have also included issues with operations attempted to manage resources or resource groups. This could have resulted in a downstream impact on other Azure services that rely on Azure Resource Manager. Preliminary Root Cause: We have identified a change to backend role instances leveraged by Azure Resource Manager causing the timeouts and is root cause of the failure. Mitigation: We mitigated background job execution systems causing failures and performed a roll back of recent change following our safe deployment practices (SDP) to return ARM to a previous healthy state, mitigating the issue. The roll back took several hours to complete globally following our SDP process. Next Steps: We will also continue to investigate to establish the full root cause and prevent future occurrences. You can stay informed about Azure service issues, maintenance events, or advisories by creating custom service health alerts (https://aka.ms/ash-videos" class="wa-link-status">https://aka.ms/ash-videos" rel="noopener noreferrer" target="_blank">https://aka.ms/ash-videos" class="wa-link-status">https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts" class="wa-link-status">https://aka.ms/ash-alerts" rel="noopener noreferrer" target="_blank">https://aka.ms/ash-alerts" class="wa-link-status">https://aka.ms/ash-alerts for how-to documentation) and you will be notified via your preferred communication channel(s).