Azure Communication Services - Multiple Regions

Update timeline

1. postmortem Aug 13, 2022, 03:30 AM UTC

   What happened? Between 18:13 UTC on 12 Aug 2022 and 03:30 UTC on 13 Aug 2022, customers using Azure Communication Services (ACS) may have experienced authentication failures, or failures using our APIs. As a result, multiple scenarios may have been impacted including SMS, Chat, Email, Voice & Video scenarios, Phone Number Management, and Teams-ACS Interop. What went wrong, and why? An Azure resource provider provides the ability for customers to create and maintain resources, in this case, for ACS. The ACS resource provider utilizes backend Cosmos DB instances for resource metadata persistence. Prior to the incident, an increased volume of data-plane related requests was made by the resource provider to the database, which met database throughput limits. At 18:00 UTC on 12 Aug 2022, to meet the increased demand of requests, the database processing capacity was increased. This change in database capacity inadvertently exposed a latent code bug for the resource provider, which resulted in a functional difference in the number of database results being returned against what could be processed by the resource provider. ACS is a globally distributed service and the metadata being retrieved was required for routing calls across different regions for the authentication process. This resulted in ACS authentication failures, and subsequently caused SMS, Chat, Voice & Video, Phone Number Management, and Teams-ACS Interop scenarios to fail. How did we respond? Automated alerting indicated several failures for different ACS API requests made by customers. We immediately investigated with multiple engineering teams, however understanding the nature of the issue took time because specific fields used for debugging Cosmos DB issues were not being logged for successful queries. Due to the service configuration, a rollback of the change to the database instance would not have been supported. Once the underlying issue was identified, we developed code fixes to resolve the issue. We validated and deployed the fix using our Safe Deployment Practices, in phases. The hotfix was fully rolled out at 03:30 UTC on 13 Aug 2022, with customers reporting successful operation shortly thereafter. How are we making incidents like this less likely or less impactful? Completed: • We've completed code updates to address the latent bug and help ensure the resource provider can process all results in similar scenarios. • We've added additional logging of backend database requests for the ACS resource provider, to ensure improved traceability in future. • We have added additional gates for database configuration updates and hardening for our processes when applying such updates. We have mirrored all production configuration templates in our pre-production environment to allow validation of configuration updates before they get deployed to production. • We have completed additional Failure Mode Analyses (FMA) across different ACS features. We have created repair items for a resilient service architecture to improve failure recovery time. How can we make our incident communications more useful? We are piloting this "PIR" format as a potential replacement for our "RCA" (Root Cause Analysis) format. You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YTYN-5T8