Carrier Automate and Call2Teams portal login and sync issues

Update timeline

1. investigating Oct 09, 2025, 10:01 AM UTC

   We are currently investigating an issue affecting our Carrier Automate and Call2Teams services across all regions. We believe this may be related to a vendor issue originating from Microsoft Azure. As a result, users may experience intermittent login issues when accessing the Carrier Automate and Call2Teams portals, along with synchronisation problems within Call2Teams. Calling functionality remains unaffected across both services and continues to operate as normal. We will provide updates every 60 minutes as further information becomes available. We apologise for any inconvenience caused and appreciate your patience during this time. Dstny Support

2. investigating Oct 09, 2025, 10:27 AM UTC

   We are continuing to investigate this issue.

3. identified Oct 09, 2025, 10:39 AM UTC

   Microsoft has now confirmed an issue within Azure, which is impacting our Carrier Automate and Call2Teams services across all regions. You can view official notifications from Microsoft via their Azure Status Page - https://azure.status.microsoft/en-us/status Our engineering teams continue to investigate and are working to mitigate the impact wherever possible. We will continue to provide updates every 60 minutes as more information becomes available. We apologise for any disruption and appreciate your continued patience. Dstny Support

4. identified Oct 09, 2025, 11:35 AM UTC

   Microsoft has confirmed ongoing recovery efforts within Azure, which continue to impact our Carrier Automate and Call2Teams services across all regions. As part of their mitigation process, Microsoft has been restarting underlying Kubernetes instances, and Azure Front Door (AFD) instances are now coming back online. Customers should begin to see service recovery, with full mitigation expected within the next 90 minutes. Our engineering teams remain engaged and are monitoring the situation closely, while continuing to assess and mitigate any residual impact where possible. We will provide a further update in 60 minutes or sooner if new information becomes available. We appreciate your continued patience and understanding. Dstny Support

5. monitoring Oct 09, 2025, 12:51 PM UTC

   Microsoft has now confirmed that 96% of impacted resources have been successfully recovered as part of their ongoing mitigation efforts within Azure. Work continues to restore the remaining 4%, and we are closely monitoring progress. From our own testing, we can confirm that navigation within the Call2Teams portal is performing smoothly, and synchronisation is operating within expected timelines. Additionally, login to the Carrier Automate portal is working as expected. While services are showing strong signs of recovery, our engineering teams remain actively engaged and will continue to monitor for any residual impact. We will provide a further update in 60 minutes or sooner should new information become available. We appreciate your continued patience and understanding. Dstny Support

6. monitoring Oct 09, 2025, 02:10 PM UTC

   Microsoft has advised that Azure customers using Azure Front Door (AFD) should now be seeing consistent availability, with slightly elevated latencies continuing to improve as recovery progresses. However, some customers may still experience intermittent issues until Microsoft confirms full resolution of the incident. We are continuing to monitor the situation closely and will provide a further update in 60 minutes. Thank you for your continued patience and understanding. Dstny Support

7. monitoring Oct 09, 2025, 04:07 PM UTC

   Microsoft has advised that Azure customers using Azure Front Door (AFD) should now be seeing consistent availability, with slightly elevated latencies continuing to decrease as recovery progresses. Our engineering teams have observed no further issues within the Carrier Automate portal, the Call2Teams portal, or with Call2Teams synchronisation. All services continue to operate as expected. We will continue to monitor the situation closely and aim to provide a further update by 08:00 UTC tomorrow, or sooner if new information becomes available. Thank you for your continued patience and understanding. Dstny Support

8. resolved Oct 10, 2025, 08:03 AM UTC

   We are pleased to confirm that this incident has now been fully resolved. Microsoft has advised that, following continuous monitoring of latency improvements, they have declared the issue mitigated after receiving confirmation of recovery from affected customers. Over the past 16 hours, our engineering teams have also observed no further issues within the Carrier Automate portal, the Call2Teams portal, or with Call2Teams synchronisation. All services continue to operate as expected. The incident was caused by an issue originating with a third-party provider (Microsoft Azure). We are actively engaging with them to gain a full understanding of the root cause and to explore steps to prevent similar occurrences in future. Please note that Microsoft’s incident timelines differ from our own. Their documentation states that they will complete an internal retrospective within 3 days and aim to publish a Post Incident Review to impacted customers within 14 days. While we will make every effort to provide a detailed post-mortem, this will be on a best-endeavours basis and dependent on receiving the necessary information from Microsoft. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. Should you have any further questions or concerns, please don’t hesitate to contact our support team. Thank you, Dstny Support

9. postmortem Oct 14, 2025, 02:01 PM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Jonathan Goodier **Date Post Mortem Completed \(UTC\)** 14th Oct 2025, 13:30 **Incident Summary** Between 07:50 UTC and 16:00 UTC on 9th October 2025, Dstny services including Carrier Automate, Call2Teams, and Call2Teams Go experienced degraded performance and intermittent errors when accessing the service portals and completing account syncs. The issue was caused by a major service degradation in Microsoft Azure Front Door \(AFD\), which is used to route traffic to these services. The incident was proactively detected via monitoring alerts and escalated internally. Microsoft confirmed the issue as a multi-region capacity loss affecting over 200 AFD environments. Partial recovery began mid-morning, and full resolution was confirmed by the following day. **Root Cause** The incident was triggered by a latent bug in the Azure Front Door data plane. During onboarding of customer workloads to a newer control plane version, erroneous metadata was created due to a software defect. A cleanup operation of this metadata exposed the latent bug, causing infrastructure crashes across multiple edge sites in Europe and Africa. This led to increased latency and timeouts as traffic was redistributed to remaining healthy nodes, which became overloaded. Dstny services were impacted due to their reliance on Azure Front Door for routing, firewalling, and load balancing. This caused slow portal responses, sync failures, and intermittent loading issues for Call2Teams Go. **Incident Resolution** Microsoft restored Azure Front Door service through a combination of automated restarts and manual intervention. Failover operations were also initiated for critical services. Dstny monitoring confirmed that partial recovery from Microsoft mitigated most customer-facing impact by midday on 9th October. Full resolution was confirmed by 16:00 UTC on 9th October. No emergency changes were required internally, and service restoration was achieved without modification to Dstny infrastructure. **Mitigative Actions** To reduce the impact of similar incidents in future: * Microsoft has fixed the control plane defect and is actively resolving the latent data plane bug \(expected completion: October 2025\). * Microsoft is improving recovery time for data plane resource exhaustion \(expected completion: March 2026\). * Dstny is reviewing failover strategies and evaluating the use of Azure Traffic Manager to provide alternative routing paths during AFD outages. * Monitoring granularity will be enhanced to better isolate third-party infrastructure issues and improve incident response time. **Pending Vendor Report** Microsoft has published a preliminary report outlining the cause and resolution of the Azure Front Door outage. A final post-incident review is expected but has not yet been scheduled. Once available, it will be reviewed in full and any additional mitigative actions will be taken as required. The final report will be published at: [https://azure.status.microsoft/en-us/status/history](https://azure.status.microsoft/en-us/status/history)