RCA - Service availability issue - Brazil South

Update timeline

1. postmortem Jul 23, 2021, 09:28 PM UTC

   SUMMARY OF IMPACT: Between 19:00 UTC and 21:28 UTC on 23 Jul 2021, a subset of customers in Brazil South may have experienced issues connecting to a number of Azure services hosted in this region, including, but not limited to, App Service, ExpressRoute, Azure Search, and VPN Gateway. ROOT CAUSE: On 23 Jul 2021, at 19:00 UTC, we received an alert from our health monitoring that there were network connectivity issues in the Brazil South region. On initial investigation, we identified a problem in the physical network infrastructure. Upon further investigation, we found that the physical network links to some of the devices within a single cluster were disrupted during a planned maintenance activity. During this maintenance activity, a technician was executing a change that impacted the incorrect set of cables. The work that was executed on site led to the disruption of services. MITIGATION: The Data Center Operations team worked at restoring some of the physical links which restored network connectivity to operational threshold by 21:20 UTC. By 21:28 UTC most of the impacted services were recovered. Data Center Operations team then continued to restore the remaining subset of cables and completed this activity at 00:52 UTC on 24 Jul 2021, to bring the services back through normal connectivity. NEXT STEPS: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): • Reviewing existing network datacenter operations cabling processes and standards. • Revisiting and retraining our data center staff on processes. PROVIDE FEEDBACK: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey