Azure Portal - Errors accessing the Azure portal

Update timeline

1. postmortem Jun 09, 2023, 05:00 PM UTC

   Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/QNPD-NC8 What happened? Between 15:10 and 17:00 UTC on 9 June 2023, customers may have experienced error notifications when trying to access the Azure Portal (portal.azure.com), and actions to load new HTTP content in the Azure Portal may have resulted in error notifications. Customers may also have experienced issues accessing other services built on the Azure Portal, like the Microsoft Entra Admin center (entra.microsoft.com) and Microsoft Intune (intune.microsoft.com). What went wrong and why? Our internal telemetry reported an anomaly with increased request rates as the Azure Portal was displaying a “service unavailable” message in multiple geographies. Traffic analysis showed an anomalous spike in HTTP requests being issued against Azure Portal origin servers, bypassing existing automatic preventive recovery measures and triggering the service unavailable response. We identified that the traffic spikes were issuing floods of incoming requests at a magnitude significantly larger than the normal traffic patterns, which overwhelmed the origin servers. Upon further investigation, we have determined the anomalous traffic spikes to be Layer 7 distributed denial-of-service (DDoS) attacks. Multiple types of methods were utilized which included HTTP request floods distributed from across the globe, and cache bypass techniques to overload the Azure Portal origin servers. For more details, including a characterization of the attack types used and recommendations for customers to protect against these, refer to this Microsoft Security Response Center (MSRC) blog post (https://msrc.microsoft.com/blog/2023/06/microsoft-response-to-layer-7-distributed-denial-of-service-ddos-attacks/). Our automatic preventive recovery measures help us mitigate an average of about 1,435 DDoS attacks per day. (More information can be found here: https://www.microsoft.com/security/blog/2023/02/21/2022-in-review-ddos-attack-trends-and-insights/). However, this attack resulted in the issues mentioned above, and we have hardened our protections from the lessons we’ve learned from it. How did we respond? We were alerted by our internal monitoring of the traffic spikes impacting the availability of the Azure Portal. Within 15 minutes, engineers from both our Portal and networking teams were engaged and began investigating the issue. DDoS protection rules were adjusted to block the traffic based on the attack pattern. Traffic throttling rules were adjusted to throttle the requests. Additional Azure Portal server instances were added, to handle increased load. Any unhealthy Azure Portal instances were rebooted, to ensure they could properly take on external traffic. Applying these mitigation steps improved Azure Portal availability. By 16:20 UTC, we were able to block 80% of the attack traffic which significantly helped limit global impact. At that point, we were at 50% Portal availability, mitigating impact for some customers. A spike was observed at 16:50 UTC, which was absorbed by the platform due to our in-built DDoS protection mechanisms and additional actions that were being taken. Our internal monitoring started reporting a healthy state back to baseline at 17:00 UTC for all Azure Portal endpoints. Our engineers continued to closely monitor the situation and carried out proactive measures to harden the system against the potential for a recurrence of the issue. How are we making incidents like this less likely or less impactful? • Making the Azure Portal more efficient so scale up runs more quickly. (Completed) • Using proactive defense logic in adjusting traffic blocking and throttling rules. (Completed) • Blocking invalid requests and server responses more aggressively. (Completed) • Increasing Azure Portal scale to cope with high demand leads more efficiently. (Completed) • Improving our configuration with additional caching mechanisms for the Azure Portal. (Estimated completion: July 2023) • Making the Azure Portal startup process faster. (Estimated completion: July 2023) • Improving our internal Azure incident management monitoring to detect such indicators of broader impact more quickly and efficiently and help to notify customers sooner. (Estimated completion: August 2023) How can customers make incidents like this less impactful? When the Azure Portal is unavailable: • To manage your Azure environment, consider using Azure Command-Line Interface (https://learn.microsoft.com/cli/azure/) and PowerShell (https://learn.microsoft.com/powershell/azure/get-started-azureps?view=azps-10.0.0). These were not impacted during this incident. • To manage resources, consider using Azure Resource Manager (ARM) REST API as well as tools that are built using API endpoints and scripting (https://learn.microsoft.com/rest/api/resources/). • To submit support requests, consider using Azure Support REST API (https://learn.microsoft.com/rest/api/support/), Command-Line Interface (https://learn.microsoft.com/cli/azure/azure-cli-support-request) or Customer Service Phone Numbers (https://go.microsoft.com/fwlink/p/?linkid=2201739). For your own Application DDoS protection: • We recommend reviewing Layer 7 protection services, such as Azure Web Application Firewall (WAF) (available with Azure Front Door, Azure Application Gateway) to help reduce the impact of Layer 7 DDoS attacks to your applications (https://learn.microsoft.com/azure/web-application-firewall/shared/application-ddos-protection). For all Azure service issues: • Finally, consider ensuring that the right people in your organization will be notified about any future service issues by configuring Azure Service Health alerts: (https://aka.ms/ash-alerts). These can trigger emails, SMS, webhooks, push notifications (via the Azure Mobile app https://aka.ms/AzureMobileApp) and more. How can we make our incident communications more useful? You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/QNPD-NC8