RCA - Azure Active Directory - Issues when attempting to authenticate

Update timeline

1. postmortem Nov 17, 2021, 02:50 AM UTC

   Summary of Impact: (Updated on 28 Sep 2022 to reflect the correct incident duration) Between 19:30 UTC and 22:10 UTC on 16 Nov 2021 and between 23:00 UTC on 16 Nov 2021 and 02:50 UTC on 17 Nov 2021, a subset of customers using Azure Active Directory B2C in the Australia region may have encountered failures while attempting to authenticate. These attempts may have succeeded on retry. Root Cause: The engineering team has determined that an underlying code defect triggered some service nodes to experience a shortage of outbound ports to complete network tasks. This in turn caused an increase in service latency and timeouts. Mitigation: As an immediate mitigation, traffic was migrated away from the affected nodes into healthy nodes in a different environment. The engineering team also proactively increased capacity in this environment. Next Steps: We sincerely apologize for the impact to affected customers. We are currently conducting an internal review to take additional steps to continuously improve the Microsoft Azure platform and our processes to ensure such issues do not occur in the future. In this case, this includes (but is not limited to): • Enhancement of service capacity and automatic scaling rules to include additional dependencies such as outbound port connections • Improvement of detection logic for cases where long-running network calls affect customer-facing performance • An overall service capacity review in the affected region, and worldwide Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey