Document Drafter Outage History

Started Oct 24, 2026, 09:53 AM UTC· resolved Oct 24, 2026, 11:02 AM UTC

1. identified Oct 24, 2026, 09:53 AM UTC

   We are currently experiencing a service disruption affecting customers who use encryption via T-Systems. The issues affects portals in the EU and in the UK. Due to an outage in T-Systems' infrastructure, it is at the moment not possible to open existing questionnaires or create new questionnaires in Document Drafter. We are in close contact with T-Systems and are following the situation closely. We will notify you as soon as the service is fully restored.

2. identified Oct 24, 2026, 09:54 AM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Oct 24, 2026, 09:55 AM UTC

   A fix has been implemented and we are monitoring the results.

4. identified Oct 24, 2026, 09:56 AM UTC

   The issue has been identified and a fix is being implemented.

5. identified Oct 24, 2026, 09:57 AM UTC

   We are waiting for T-systems to restore services

6. identified Oct 24, 2026, 11:01 AM UTC

   We are continuing to work on a fix for this issue.

7. resolved Oct 24, 2026, 11:02 AM UTC

   T-System appears to be back on line and we do not see issues anymore. We will oontinue moitoring and update later with a post mortem.

Read the full incident report →

Started Aug 03, 2026, 04:37 PM UTC· resolved Aug 04, 2026, 04:57 AM UTC

1. investigating Aug 03, 2026, 04:37 PM UTC

   We are currently investigating routing problems in Azure Front Door

2. investigating Aug 03, 2026, 05:04 PM UTC

   Microsoft is monitoring the issue. We will update again within 60 minutes.

3. investigating Aug 03, 2026, 06:11 PM UTC

   We are continuing to investigate this issue.

4. investigating Aug 03, 2026, 06:12 PM UTC

   The issue appears to have been resolved. We will continue to monitor and provide a post mortem here in due course.

5. monitoring Aug 03, 2026, 07:23 PM UTC

   Some edge access points for AFD was affected, Microsoft engineers has apply changes, and we are now monitoring the flow of data.

6. resolved Aug 04, 2026, 04:57 AM UTC

   Monitoring has completed, no services interrupted

Read the full incident report →

Started Oct 29, 2025, 04:00 PM UTC· resolved Oct 30, 2025, 06:54 AM UTC

1. investigating Oct 29, 2025, 04:00 PM UTC

   We are currently experiencing degraded response times originating from an issue with our Azure infrastructure. This latency is causing the service to be unavailable or partially unavailable for some users. Our engineering team is actively investigating the root cause and working toward remediation. We will provide the next status update in no more than 1 hour.

2. investigating Oct 29, 2025, 04:41 PM UTC

   Update from Microsoft: Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal and services. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or soon.

3. investigating Oct 29, 2025, 05:19 PM UTC

   New update from Microsoft; We have failed the portal away from Azure Front Door (AFD) to mitigate the portal access issues. Customers should be able to access the Azure management portal directly. We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update. This message was last updated at 17:14 UTC on 29 October 2025

4. investigating Oct 29, 2025, 05:43 PM UTC

   We are continuing to investigate this issue.

5. investigating Oct 29, 2025, 05:43 PM UTC

   We have a new update from Microsoft We are taking several concurrent actions: Firstly, where we are blocking all changes to the AFD services, this includes customer configuration changes as well. At the same time, we are rolling back our AFD configuration to our last known good state. As we rollback we want to ensure that the problematic configuration doesn't re-initiate upon recovery. We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update. This message was last updated at 17:40 UTC on 29 October 2025

6. investigating Oct 29, 2025, 06:22 PM UTC

   We are continuing to investigate this issue.

7. investigating Oct 29, 2025, 06:22 PM UTC

   Update from Microsoft: We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes. We do not have an ETA for full mitigation, we will update this communication within 30 minutes, once the deployment is completed.

8. investigating Oct 29, 2025, 07:12 PM UTC

   Services are back online except RestAPI and Power Automate, We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement. We will keep you updated.

9. investigating Oct 29, 2025, 07:26 PM UTC

   As to Rest API and a small sub-set of customers still affected, we have the following update from Microsoft: At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

10. investigating Oct 30, 2025, 06:46 AM UTC

    On October 29, 2025, a global issue affecting Azure Front Door (AFD) caused service disruptions across multiple regions. Our team implemented manual re-routing measures and successfully restored approximately 98% of our services by 9:00 PM CET on October 29. Microsoft resolved the underlying Azure Front Door problem at 1:00 AM CET on October 30, and all systems have since been fully operational. We are closely monitoring performance and awaiting the official Post-Incident Report from Microsoft Azure to better understand the root cause and ensure long-term resiliency improvements. Thank you for your patience and understanding during this incident.

11. resolved Oct 30, 2025, 06:54 AM UTC

    This incident has been resolved.

12. postmortem Nov 11, 2025, 08:03 AM UTC

    We successfully mitigated the impact of the Azure Front Door \(AFD\) incident that affected several Azure services from October 29 to October 30, 2025. Our immediate action was to **manually move our services away from Azure Front Door/Content Delivery Network \(CDN\)**, which restored the availability of the Document Drafter application. While our application remained available and customer data integrity was maintained, we are taking further steps to enhance resilience. We are currently **updating our disaster recovery plans** to formalize failover strategies for all external Azure components, and we are **reviewing our use of Microsoft's Content Delivery Network** to ensure robust availability. We are also tracking Microsoft's planned repairs, which include implementing extended bake times, removing asynchronous processing, and enhancing testing for their configuration rollout pipeline. Please find Microsoft’s post mortem below.

Read the full incident report →

Started Oct 09, 2025, 08:26 AM UTC· resolved Oct 09, 2025, 05:07 PM UTC

1. investigating Oct 09, 2025, 08:26 AM UTC

   We are experiencing issues with Azure DNS resolution which may cause partial outage in some portal instances. We are investigating the issue and will update.

2. investigating Oct 09, 2025, 09:20 AM UTC

   We are continuing to investigate the issue. There are large outages on Microsoft Azure's side. We will keep you updated as soon as we know more. We apologies for the interruption. To check health on Azure: https://statusgator.com/services/azure

3. investigating Oct 09, 2025, 10:11 AM UTC

   We are continuing to investigate this issue.

4. identified Oct 09, 2025, 10:12 AM UTC

   Microsoft has now reported that there is an outage and global issue, you can check the same updated on Azure's Status page: Impact Statement: Starting at 07:40 UTC on 09 October 2025, Azure customers using Azure Front Door (AFD) may experience intermittent issues when accessing their services, this includes the ability to access the Azure Portal. Current Status: Our monitoring detected a significant capacity loss, of about 30% of Azure Front Door instances predominantly across Europe and Africa. At this stage we have ruled out any deployments which could have triggered this event. The next update will be provided within 60 minutes, or as events warrant. This message was last updated at 10:14 UTC on 09 October 2025 https://azure.status.microsoft/en-us/status We will keep updating you.

5. identified Oct 09, 2025, 10:18 AM UTC

   We are continuing to work on a fix for this issue.

6. identified Oct 09, 2025, 11:03 AM UTC

   We are continuing to work on a fix for this issue.

7. identified Oct 09, 2025, 11:04 AM UTC

   We are continuing to work on a fix for this issue. We have received this update from Microsoft: Current Status: Our monitoring detected a significant capacity loss of about 30% of Azure Front Door instances, predominantly across Europe, Middle East, and Africa. We understand that this is due to a dependency on some underlying Kubernetes instances that crashed. We have ruled out any deployments that could have triggered this event. We have been restarting these underlying Kubernetes instances, and AFD instances are coming back online. Customers should start seeing recovery as we bring these instances back online, and we expect full mitigation within the next 90 minutes. The next update will be provided within 60 minutes. This message was last updated at 11:01 UTC on 09 October 2025

8. identified Oct 09, 2025, 11:40 AM UTC

   We are continuing to work on a fix for this issue and have a new update from Microsoft. Impact Statement: Starting at 07:40 UTC on 09 October 2025, Azure customers using Azure Front Door (AFD) may experience intermittent delays or timeouts when accessing their services. This includes the ability to access the Azure Portal and the Entra Admin Portal. Current Status: As we continue our mitigation efforts, we have successfully recovered 96% of impacted resources, teams are working on recovering the remaining 4% of impacted customers. The next update will be provided within 60 minutes. This message was last updated at 11:36 UTC on 09 October 2025

9. identified Oct 09, 2025, 02:26 PM UTC

   We have another update from Microsoft. Issue should be all but resolved. Impact Statement: Starting at 07:40 UTC on 09 October 2025, Azure customers using Azure Front Door (AFD) from EMEA, Middle East and Africa may experience intermittent delays or timeouts when accessing their services. Current Status: Azure customers using Azure Front Door (AFD) should see consistent availability with slightly higher latencies decreasing as we continue to recover additional resources. The next update will be provided within 60 minutes. This message was last updated at 14:05 UTC on 09 October 2025

10. identified Oct 09, 2025, 04:09 PM UTC

    We are continuing to work on a fix for this issue and have a new update from Microsoft. Impact Statement: Starting at 07:40 UTC on 09 October 2025, Azure customers using Azure Front Door (AFD) from EMEA, Middle East and Africa may experience intermittent delays or timeouts when accessing their services. Current Status: Azure customers using Azure Front Door (AFD) should see consistent availability with slightly higher latencies decreasing as we continue to recover additional resources. The next update will be provided within 60 minutes. This message was last updated at 16:03 UTC on 09 October 2025

11. identified Oct 09, 2025, 05:06 PM UTC

    Microsoft has announced that the issue is resolved. We will communicate once we obtain a post mortem.

12. resolved Oct 09, 2025, 05:07 PM UTC

    This incident has been resolved.

13. postmortem Oct 30, 2025, 07:25 AM UTC

    We have received the following preliminary post mortem from Microsoft. ‌ We have received the following preliminary post mortem from Microsoft. COMMUNICATION: _Join one of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident \(to hear from our engineering leaders, and to get any questions answered by our experts\) or watch a recording of the livestream \(available later, on YouTube\):_ [_https://aka.ms/AIR/QNBQ-5W8_](https://aka.ms/AIR/QNBQ-5W8) **What happened?** Between 07:50 UTC and 16:00 UTC on 09 October 2025, Microsoft services and Azure customers leveraging Azure Front Door \(AFD\) and Azure Content Delivery Network \(CDN\) may have experienced increased latency and/or timeouts – primarily across Africa and Europe, as well as Asia Pacific and the Middle East. This impacted the availability of the Azure Portal as well as other management portals across Microsoft. Peak failure rates for AFD reached approximately 17% in Africa, 6% in Europe, and 2.7% in Asia Pacific and the Middle East. Availability was restored by 12:50 UTC, though some customers continued to experience elevated latency. Latency returned to baseline levels by 16:00 UTC, at which point the incident was mitigated. **What do we know so far?** AFD routes traffic using globally distributed edge sites and supports Microsoft services including the management portals. The AFD control plane generates system metadata that the data plane consumes for customer-initiated ‘create’, ‘update’, or ‘delete’ operations on AFD or CDN profiles. One of the trigger conditions for this incident was a software defect in the latest version of the AFD control plane which had been rolled out six weeks prior to the incident, in line with our safe deployment practices. Newly created customer tenant profiles were being onboarded to the newer control plane version. Our service monitoring detected elevated data plane crashes due to a previously unknown bug – triggered by erroneous metadata, generated by a particular sequence of profile update operations. Our automated protection layer intercepted this in early update stages and prevented this metadata from propagating any further to the data plane, thereby averting any customer impact at that time. In addition, as the newer control plane was running in tandem with the previous version of the control plane, we disabled the new control plane from taking any requests. On 09 October 2025, we initiated a cleanup of the affected tenant configuration with the erroneous metadata. Since the automated protection system was blocking the impacted customer tenant profile updates in the initial stage, we temporarily bypassed it to allow the cleanup of the tenant configuration to proceed. By bypassing the protection system, the erroneous metadata was inadvertently able to propagate to later stages – and triggered the bug in the data plane that crashed the data plane service. This resulted in a disruption to a significant number of edge sites across Europe and Africa, approximately 26% of AFD data plane infrastructure resources in these regions were impacted. As part of AFD mechanisms to manage traffic, load was automatically distributed to nearby edge sites \(including in Asia Pacific and the Middle East\). Additionally, as regional business hours traffic started ramping up, it added to the overall traffic load. The increased volume of traffic on the remaining healthy edge sites resulted in high resource utilization, which exceeded operational thresholds. This triggered an additional layer of protection which started distributing traffic to a broader set of edge sites globally, to reduce further impact. Recovery required a combination of automated restarts, manual intervention where automated restarts were taking too long, and traffic failover operations for impacted management portals. Full mitigation was achieved once edge site infrastructure resources stabilized and latency returned to normal. Additionally, initial customer notifications were delayed primarily due to challenges determining impact, while attempting to target communications to those impacted. We have automated communications to notify customers of incidents quickly, unfortunately this capability was not yet supported in this incident scenario. **How did we respond?** * 07:30 UTC on 09 October 2025 – The cleanup operation was initiated. * 07:50 UTC on 09 October 2025 – Initial customer impact began, and increased over the next 90 minutes. * 08:13 UTC on 09 October 2025 – Our telemetry detected resource availability loss across multiple AFD edge sites. We began investigating as impact continued to grow. * 09:04 UTC on 09 October 2025 – We identified that the crashes were due to the previously identified data plane bug. * 09:08 UTC on 09 October 2025 – Automated restarts began for our AFD infrastructure resources, and manual intervention began for resources that did not recover automatically. * 09:15 UTC on 09 October 2025 – Customer impact had grown to be at its peak. * 10:01 UTC on 09 October 2025 – Communications were published to the Azure Status page. * 10:45 UTC on 09 October 2025 – Targeted customer communications were sent to Azure Service Health. * 11:59 UTC on 09 October 2025 – Management portals, like the Azure Portal, performed failover operations \(including using scripts to update the load balancing configuration, to split traffic between multiple routes\) helping restore its service availability. * 12:50 UTC on 09 October 2025 – Availability for AFD fully recovered, however a subset of customers may still have been experiencing elevated latency. * 16:00 UTC on 09 October 2025 – After continuous monitoring of latency improvement, we declared the incident as mitigated after confirming recovery. **How are we making incidents like this less likely or less impactful?** * We have hardened our standard operating procedures, to ensure that the configuration protection system is not bypassed for any operation. \(Completed\) * We have fixed the control plane defect which generated the erroneous tenant metadata that led to the data plane resource crashes. \(Completed\) * We have fixed the bug in the data plane. \(Completed\) * We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation. \(Estimated completion: November 2025\) * We are making improvements to our Azure Portal failover systems from AFD, to be more robust and automated. \(Estimated completion: December 2025\) * We are building additional runtime configuration validation pipelines against a replica of real-time data plane, as a pre-validation step prior to applying them broadly. \(Estimated completion: March 2026\) * We are improving data plane resource instance recovery time, following any impact to the data plane. \(Estimated completion: March 2026\) **How can customers make incidents like this less impactful?** * Consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: [https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview](https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview) * Consider reviewing our best practices for Azure Front Door architecture: [https://learn.microsoft.com/azure/well-architected/service-guides/azure-front-door](https://learn.microsoft.com/azure/well-architected/service-guides/azure-front-door) * Consider implementing retry patterns with exponential backoff, to improve workload resiliency: [https://learn.microsoft.com/azure/architecture/patterns/retry](https://learn.microsoft.com/azure/architecture/patterns/retry) * More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: [https://aka.ms/AzPIR/WAF](https://aka.ms/AzPIR/WAF) * The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: [https://aka.ms/AzPIR/Monitoring](https://aka.ms/AzPIR/Monitoring) * Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: [https://aka.ms/AzPIR/Alerts](https://aka.ms/AzPIR/Alerts) **How can we make our incident communications more useful?** You can rate this PIR and provide any feedback using our quick 3-question survey: [http://aka.ms/AzPIR/QNBQ-5W8](http://aka.ms/AzPIR/QNBQ-5W8)

Read the full incident report →

Started Apr 01, 2025, 09:48 AM UTC· resolved Apr 01, 2025, 11:10 AM UTC

1. investigating Apr 01, 2025, 09:48 AM UTC

   Due to a global outage affecting Azure Cosmos DBs critical to the platform, the Document Drafter is currently out of service. We are working with Microsoft to get back online as soon as possible. We will keep you updated.

2. investigating Apr 01, 2025, 09:48 AM UTC

   We are continuing to investigate this issue.

3. monitoring Apr 01, 2025, 09:57 AM UTC

   The Document Drafter platform is back online. Microsoft has informed that there was an outage of their global CosmosDB gateway. Microsoft has now restored the services and will continue to monitor the affected components. We will revert with further information as it becomes available.

4. monitoring Apr 01, 2025, 11:09 AM UTC

   We are continuing to monitor for any further issues.

5. resolved Apr 01, 2025, 11:10 AM UTC

   Microsoft has advised that the issue is now fully resolved. Between 08:51 and 10:15 UTC on 01 April 2025, Microsoft identified customer impact resulting from a power event in the North Europe region which impacted Storage, CosmosDB, and various other resources which are not used in Document Drafter. Microsoft has confirmed that all affected services have now recovered. A power maintenance event led to temporary power loss in a single datacenter, in Physical Availability Zone 2, in the North Europe region affecting multiple racks and devices. Microsoft has advised that power has been fully restored and services are seeing full recovery. We will now evaluate internally as per our SOPs. We will not provide more information unless you request it.

Read the full incident report →