Dstny Outage History

Started Nov 26, 2025, 01:54 PM UTC· resolved Nov 27, 2025, 04:20 PM UTC

1. investigating Nov 26, 2025, 02:13 PM UTC

   We are currently investigating a potential incident affecting Core services in the DSTNY FOX region. Our teams are actively working to identify the scope of the issue and are addressing a related network routing problem impacting access to Core services and associated applications. We will provide updates every 60 minutes as we gather more information. Thank you for your patience as we work to address this matter. Dstny Support

2. monitoring Nov 26, 2025, 02:47 PM UTC

   We are writing to inform you that an issue occurred earlier affecting Core services in the DSTNY FOX region. The incident has been fully resolved, and services have been restored. This may have caused issues with setting up and receiving inbound calls for users during the affected period. User registrations were not impacted. A change was made earlier today as part of routine maintenance, which we suspect may have been the cause of the issue. Services have been restored following a rollback of this change. We will continue to monitor the situation for the next 24 hours to ensure there is no further impact. Thank you for your understanding, and we appreciate your continued patience. If you have any questions or concerns, please don’t hesitate to contact our support team. Dstny Support

3. resolved Nov 27, 2025, 04:20 PM UTC

   We are pleased to confirm that this incident has been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. We have determined the root cause and have implemented measures to prevent future occurrences. To provide transparency and insight into the incident, a detailed post-mortem report will be made available within the next 5 business days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. Should you have any further questions or concerns, please feel free to reach out to our support team. Thank you, Dstny Support

4. postmortem Dec 04, 2025, 09:58 AM UTC

   **Major Incident Category** Service Outage **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 03 Dec 2025, 10:35 **Incident Summary** On 26th November 2025, at 13:44 UTC, customers using the DSTNY SE FOX Core service experienced a temporary outage affecting all new inbound and outbound calls. The issue was detected proactively through our alerting platform. Recovery was achieved at 13:50 UTC by promptly rolling back a recent configuration change, restoring normal service for all affected customers. Calls already in progress were not impacted during the incident. **Root Cause** The incident was caused by a configuration change intended to improve network access. During this change, the network link between two data centres was taken out of service, resulting in a loss of connectivity between them. This link failure went unnoticed at the time because there was insufficient monitoring and alerting for link failures between the data centres. The issue was later identified through alerts generated by the Core alerting platform. This alerting functioned as expected and is separate from the missing monitoring and alerting on the inter‑datacentre link. As a result, when the configuration change was applied and the network link went down, connectivity was lost and call services were disrupted. **Incident Resolution** The issue was resolved promptly by rolling back the recent configuration change, which restored the previous network settings and re-established connectivity. This action reinstated normal service for all affected customers within six minutes of the initial disruption. **Mitigative Actions** * Implementation of enhanced monitoring and alerting for critical network connections to ensure any future failures are immediately detected. * Review and improvement of change management procedures to ensure all changes are subject to appropriate testing and risk assessment, even for standard maintenance. * Additional training and awareness for operational teams regarding the importance of monitoring and pre-change checks.

Read the full incident report →

Started Nov 25, 2025, 04:07 PM UTC· resolved Nov 27, 2025, 04:19 PM UTC

1. investigating Nov 25, 2025, 05:26 PM UTC

   We are currently investigating a potential incident affecting Call2Teams sync jobs. This issue impacts all regions and affects customers’ ability to successfully sync their Call2Teams accounts, which may prevent changes to users or onboarding new users during this time. Please note: this incident impacts sync only and does not affect calls or registrations. As part of our mitigation efforts, we have temporarily paused sync jobs to alleviate pressure on the system caused by an unusually high number of concurrent sync jobs today. During this time, queued sync jobs will appear in a cancelled state. We will provide updates every 60 minutes as we gather more information. Thank you for your patience as we work to address this matter. Dstny Support

2. investigating Nov 25, 2025, 05:27 PM UTC

   We are currently investigating an issue affecting Call2Teams sync jobs across all regions. This is causing delays and cancellations of sync processes, which may prevent changes to users or onboarding new users during this time. Please note: this incident impacts sync only and does not affect calls or registrations. Our teams are working to identify the root cause and implement a resolution. Sync jobs remain paused, and queued jobs will automatically cancel after 15 minutes. In-progress syncs are being allowed to complete while we address performance issues with the platform. We are also exploring options to temporarily scale the database and may manually fail selected syncs once system access improves. Updates will be provided every 60 minutes as we learn more. We apologise for any inconvenience caused and appreciate your patience during this time. Dstny Support

3. monitoring Nov 25, 2025, 06:23 PM UTC

   We are writing to inform you that an issue occurred earlier affecting Call2Teams sync jobs across all regions. The incident has now been fully resolved, and services have been restored. This may have caused delays or cancellations of sync processes, preventing changes to users or onboarding new users during the affected period. Our engineering team has manually terminated all existing sync jobs and reopened channels for new jobs. Testing confirms that new sync requests are processing successfully. Please note that some previously queued jobs may still auto-cancel due to the steps taken during recovery. We will continue to monitor the situation for the next 24 hours to ensure there is no further impact. Thank you for your understanding, and we appreciate your continued patience. If you have any questions or concerns, please don’t hesitate to contact our support team. Dstny Support

4. identified Nov 26, 2025, 03:52 PM UTC

   We are writing to inform you that after a period of monitoring, the Call2Teams sync platform has come under heavy load again. As a precaution, new sync jobs remain paused to stabilise the system. Customers may see some sync requests automatically cancelled during this time. Our platform engineering team is now implementing mitigation measures to address the load and restore normal operation. Another update will be provided in approximately one hour. Thank you for your patience while we work to resolve this matter. Dstny Support

5. identified Nov 26, 2025, 05:35 PM UTC

   We are continuing to investigate the issue affecting Call2Teams sync jobs. Following recent measures to improve performance, we are experiencing further connectivity challenges, and sync jobs remain paused while our engineering team works to restore stability. We have identified the underlying issue and are actively working on a resolution. Customers may continue to see cancellations during this time. A further update will be provided in approximately one hour. Thank you for your patience while we work to resolve this matter. Dstny Support

6. monitoring Nov 26, 2025, 06:57 PM UTC

   We are writing to inform you that an issue occurred earlier affecting Call2Teams sync jobs across all regions. The incident has now been fully resolved, and services have been restored. This may have caused delays or cancellations of sync processes, preventing changes to users or onboarding new users during the affected period. Our engineering team has confirmed that syncs are now processing normally. Please note that some previously queued jobs may still auto-cancel due to steps taken during recovery. We will continue to monitor the situation for the next 24 hours to ensure there is no further impact. Thank you for your understanding, and we appreciate your continued patience. If you have any questions or concerns, please don’t hesitate to contact our support team. Dstny Support

7. resolved Nov 27, 2025, 04:19 PM UTC

   We are pleased to confirm that this incident has been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. We have determined the root cause and have implemented measures to prevent future occurrences. To provide transparency and insight into the incident, a detailed post-mortem report will be made available within the next 5 business days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. Should you have any further questions or concerns, please feel free to reach out to our support team. Thank you, Dstny Support

8. postmortem Dec 02, 2025, 09:58 AM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 01 Dec 2025, 14:30 **Incident Summary** Between 25th November 2025 at 15:00 UTC and 26th November 2025 at 19:00 UTC, the Call2Teams platform experienced a major incident affecting the system responsible for processing customer data. During this period, the platform was unable to handle increased demand, resulting in significant delays and failures in synchronising user information. As a result, customers were unable to complete account changes or successfully synchronise user accounts. It is important to note that voice calling services for existing users continued to operate normally and were not impacted by this incident. The incident was fully resolved by 26th November 2025 at 19:00 UTC, with all synchronisation services restored to normal operation. **Root Cause** The incident was triggered by an overload of the database supporting user synchronisation. The database was unable to efficiently process the increased volume of requests, leading to delays and failures in synchronising user information across the platform. **Incident Resolution** To resolve the issue, we increased system capacity and restarted key services. These actions restored normal operation for all customers and resolved the synchronisation delays and failures. **Mitigative Actions** * Increased system capacity to better accommodate periods of high demand. * Optimised the performance of the synchronisation process. * Introduced enhanced monitoring and alerting to detect and address similar issues more rapidly.

Read the full incident report →

Started Oct 29, 2025, 04:15 PM UTC· resolved Oct 30, 2025, 09:04 AM UTC

1. investigating Oct 29, 2025, 04:15 PM UTC

   We are aware of an issue where the Call2Teams Admin Portal & Carrier Automate Portal is currently unreachable. Users may see the following browser message when attempting to access the site: “This site can’t be reached. admin.call2teams.com took too long to respond.” Administrators may be unable to log in or manage customer settings within the portal. This issue does not impact existing user call functionality or service availability for connected users.

2. investigating Oct 29, 2025, 04:21 PM UTC

   We are continuing to investigate this issue.

3. identified Oct 29, 2025, 04:36 PM UTC

   We have identified that the issue affecting access to the Admin Portals is related to a wider Microsoft Azure service disruption. Microsoft has reported the following: “We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.” Microsoft Azure Status Portal: https://azure.status.microsoft/en-gb/status

4. monitoring Oct 29, 2025, 07:34 PM UTC

   Microsoft has confirmed that the deployment of their ‘last known good’ configuration for Azure Front Door (AFD) has now been completed, and services are showing initial signs of recovery. Current Situation - Microsoft is restoring healthy nodes and routing traffic through stable endpoints. - Customers may see gradual improvement as recovery continues. - Configuration changes to Azure Front Door remain temporarily blocked during ongoing mitigation. - Access to the Azure Portal has been largely restored, though a few extensions (such as Marketplace) may still load intermittently. Impact to Call2Teams Access to the Admin Portals are gradually improving as Azure services recover. There remains no impact to live user call services or integrations.

5. monitoring Oct 29, 2025, 10:14 PM UTC

   We believe that access to the Admin Portal has now returned to normal operation following the earlier Microsoft Azure Front Door incident. All core portal functions are operational. However, a small number of users may continue to experience intermittent slowness or degraded performance as Microsoft finalises recovery actions across their infrastructure. We will continue to monitor the platform over the next 24 hours to ensure full stability. For complete details on the underlying Azure service incident, please visit the official Microsoft Azure Status

6. resolved Oct 30, 2025, 09:04 AM UTC

   This incident has been resolved.

7. postmortem Dec 02, 2025, 10:04 AM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 05 Nov 2025, 14:00 **Incident Summary** On 29th October 2025, at approximately 15:50 UTC, customers began experiencing issues accessing the Call2Teams and Carrier Automate portals, as well as the Call2Teams Go application. Monitoring systems promptly detected the incident, which resulted in customers being unable to access these portals or use related APIs, thereby preventing provisioning changes. Core calling functionality was not affected. Microsoft initiated service restoration by implementing a configuration update globally at 18:30 UTC. Full service was restored by 00:05 on 30th October, when Microsoft confirmed that the issue had been resolved for all customers. **Root Cause** The incident was caused by a global issue affecting Microsoft Azure Front Door. A defect in Azure Front Door’s validation process allowed a faulty configuration to bypass safety checks and propagate across the platform. This resulted in widespread node failures and disrupted service availability, leading to the loss of access to the affected portals and APIs in addition to the Call2Teams Go front end. **Incident Resolution** Resolution of the incident was managed by Microsoft. Recovery involved reloading configurations across Azure Front Door nodes and gradually rebalancing traffic to restore stability and prevent overload. This phased approach ensured the system returned to full operation and mitigated the risk of recurrence. No direct action was required by Dstny to resolve the issue. **Mitigative Actions** * Improvements to Azure Service Health alerting to ensure earlier notification of similar issues in the future. * Ongoing collaboration with Microsoft to monitor and address third-party service reliability. * Continued review and enhancement of monitoring systems to detect and respond to external service disruptions promptly.

Read the full incident report →

Started Oct 23, 2025, 09:41 AM UTC· resolved Oct 24, 2025, 04:08 PM UTC

1. investigating Oct 23, 2025, 09:41 AM UTC

   Dear Customer, We are currently investigating an issue affecting the SMP platform in the EU West and Nordics regions. This is resulting in users being unable to edit or modify ACD numbers (such as adding or removing members), as well as creating new ACD groups. Our technical teams are actively working to identify the root cause and implement a resolution. We will provide updates every 60 minutes as more information becomes available. We apologise for any inconvenience this may cause and appreciate your patience while we work to resolve the issue. Kind regards, Dstny Support

2. investigating Oct 23, 2025, 10:59 AM UTC

   Following the rollout of SMP version 2.4.1, we would like to clarify that the issues are not limited to the previously reported inability to edit or modify ACD numbers (such as adding or removing members), or to create new ACD groups. We are currently seeing a broader range of problems being reported, indicating more general issues with the platform rather than a single, isolated fault. The impact is ongoing and our teams are actively investigating. If you or your customers experience any issues, please report them to Support so they can be reviewed and addressed accordingly. We will provide a further update within the next hour. Kind regards, Dstny Support

3. investigating Oct 23, 2025, 12:01 PM UTC

   We are continuing to investigate the various issues reported following the rollout of SMP version 2.4.1. At this stage, the problems appear to be broader than initially identified, and we are working to understand the full scope. If you or your customers experience any issues, please continue to report them to Support. Each report helps us build a clearer picture and will assist in resolving the problems once they are fully identified. We appreciate your patience and will provide a further update shortly. Kind regards, Dstny Support

4. identified Oct 23, 2025, 12:40 PM UTC

   We are continuing to investigate the issues affecting SMP version 2.4.1. As part of our recovery plan, we will be rolling back to the previous version (2.4.0.1), starting with UAT and then progressing to Production. We will keep you informed as this process moves forward and will provide further updates as soon as they are available. Kind regards, Dstny Support

5. identified Oct 23, 2025, 01:37 PM UTC

   The rollback to version 2.4.0.1 in UAT has now been completed. We are currently testing to confirm whether the rollback has been successful. Initial findings in UAT indicate that some issues may still be present, and our teams are continuing to investigate. We will keep you informed as progress continues and provide further updates as soon as possible. Kind regards, Dstny Support

6. identified Oct 23, 2025, 01:53 PM UTC

   The rollback to version 2.4.0.1 in UAT has been completed and confirmed to be working successfully without issue. We are now proceeding with the rollback of the Production environment, effective immediately. Once the rollback is complete, we will carry out the necessary verifications to ensure stability and functionality. A further notice will be sent once these checks are concluded. If you experience any issues following the rollback, please report them to Support so they can be investigated promptly. Kind regards, Dstny Support

7. monitoring Oct 23, 2025, 02:19 PM UTC

   We can confirm that the rollback to version 2.4.0.1 in the Production environment has been successfully completed. All necessary verifications and checks have been carried out, and the platform is now operating as expected without issue. If you or your customers experience any problems following the rollback, please raise them with Support so they can be investigated. However, we do not anticipate any further issues, as version 2.4.0.1 was the last known stable release. We will continue to monitor service availability for the next 24 hours and do not anticipate any further impact at this time. Thank you for your continued patience and cooperation. Dstny Support

8. resolved Oct 24, 2025, 04:08 PM UTC

   We are pleased to confirm that this incident has now been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. As part of the resolution, we have rolled back to SMP version 2.4.0.1, which has been confirmed as stable. The root cause has been identified, and appropriate measures have been implemented to prevent similar issues in future. We will be rescheduling the release of version 2.4.1 once further validation is complete, and new communications will be shared in due course. To provide transparency and insight, a detailed post-mortem report will be made available within the next five working days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. If you have any further questions or concerns, please don’t hesitate to contact our support team. Thank you, Dstny Support

9. postmortem Nov 03, 2025, 04:50 PM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Jonathan Goodier **Date Post Mortem Completed \(UTC\)** 30th Oct 2025, 01:00 **Incident Summary** On 22nd October 2025 at 11:00 UTC, a service issue began affecting the ability to create or update call routing configurations, including Automatic Call Distribution \(ACD\) groups and Functional Numbers. This issue impacted all regions and was caused by a recent system update. While existing configurations continued to operate normally, attempts to modify or add new ones failed for users with specific billing configurations. The issue was detected shortly after deployment and resolved by 23rd October 2025 at 11:55 UTC. **Root Cause** The issue was caused by a change introduced during a recent system update, which altered how billing information was processed. Specifically, the system began sending an empty value instead of a null value when no billing ID was present. This led to errors in certain backend services that could not process the empty value correctly. The change was intended to support new functionality but was based on a misinterpretation during testing. **Incident Resolution** The issue was resolved by rolling back to a previous stable version of the system. This restored normal functionality for all affected users. A permanent fix is being developed to ensure the correct handling of billing information, and thorough testing will be conducted to validate compatibility across all relevant systems. **Mitigative Actions** * Update the system to send null values instead of empty strings when no billing ID is present. * Improve test coverage to include scenarios involving billing ID handling across different configurations. * Review and clarify expected behaviour for billing ID processing in documentation and training materials.

Read the full incident report →

Started Oct 20, 2025, 10:00 AM UTC· resolved Oct 21, 2025, 09:31 AM UTC

1. monitoring Oct 20, 2025, 10:00 AM UTC

   Our Platform team has identified the root cause of the issue and implemented corrective measures to restore application services. We will continue to monitor service availability over the next 24 hours, and at this time, we do not anticipate any further impact. Thank you for your patience. Dstny Support

2. resolved Oct 21, 2025, 09:31 AM UTC

   We are pleased to confirm that this incident has now been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. We have identified the root cause and implemented measures to prevent future occurrences. To provide transparency and insight into the incident, a detailed post-incident report will be made available within the next five working days. We sincerely apologise for any inconvenience this may have caused and thank you for your patience and understanding throughout the incident. If you have any further questions or concerns, please do not hesitate to contact our support team. Kind regards, Dstny Support

3. postmortem Nov 03, 2025, 05:09 PM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Jonathan Goodier **Date Post Mortem Completed \(UTC\)** 27th Oct 2025, 11:40 **Incident Summary** On 20th October 2025, between 06:50 and 09:51 UTC, users in the SE Legacy region experienced access issues when attempting to use the SMP Legacy platform. The disruption was caused by certificate errors, which prevented successful connections to the service. The issue was initially reported through support channels and confirmed by internal monitoring and additional user reports. The incident was escalated and declared a major event for investigation and resolution. **Root Cause** The incident was caused by the failure of a multi-name security certificate to renew. This occurred because a DNS record required for the renewal process had been removed. The legacy system configuration; originally designed for a smaller customer base, allowed DNS changes without safeguards, introducing vulnerabilities in certificate management. This setup was not suited to the current scale and complexity of the environment. **Incident Resolution** Resolution involved reviewing all DNS records associated with the affected certificate to ensure they correctly pointed to the certificate authorisation endpoint. Approximately 10–15 entries were verified, and incorrect or missing records were identified and corrected. Services were then restarted to trigger renewal of the failed certificate, successfully restoring access to the SMP Legacy platform. **Mitigative Actions** * Deprecation of legacy Tools Portal and API associated with SMP Legacy * Redesign of certificate management processes to eliminate reliance on multi-name certificates * Implementation of safeguards to prevent DNS changes from impacting certificate renewals * Enhanced monitoring and alerting for certificate expiry and DNS integrity

Read the full incident report →

Started Oct 09, 2025, 10:01 AM UTC· resolved Oct 10, 2025, 08:03 AM UTC

1. investigating Oct 09, 2025, 10:01 AM UTC

   We are currently investigating an issue affecting our Carrier Automate and Call2Teams services across all regions. We believe this may be related to a vendor issue originating from Microsoft Azure. As a result, users may experience intermittent login issues when accessing the Carrier Automate and Call2Teams portals, along with synchronisation problems within Call2Teams. Calling functionality remains unaffected across both services and continues to operate as normal. We will provide updates every 60 minutes as further information becomes available. We apologise for any inconvenience caused and appreciate your patience during this time. Dstny Support

2. investigating Oct 09, 2025, 10:27 AM UTC

   We are continuing to investigate this issue.

3. identified Oct 09, 2025, 10:39 AM UTC

   Microsoft has now confirmed an issue within Azure, which is impacting our Carrier Automate and Call2Teams services across all regions. You can view official notifications from Microsoft via their Azure Status Page - https://azure.status.microsoft/en-us/status Our engineering teams continue to investigate and are working to mitigate the impact wherever possible. We will continue to provide updates every 60 minutes as more information becomes available. We apologise for any disruption and appreciate your continued patience. Dstny Support

4. identified Oct 09, 2025, 11:35 AM UTC

   Microsoft has confirmed ongoing recovery efforts within Azure, which continue to impact our Carrier Automate and Call2Teams services across all regions. As part of their mitigation process, Microsoft has been restarting underlying Kubernetes instances, and Azure Front Door (AFD) instances are now coming back online. Customers should begin to see service recovery, with full mitigation expected within the next 90 minutes. Our engineering teams remain engaged and are monitoring the situation closely, while continuing to assess and mitigate any residual impact where possible. We will provide a further update in 60 minutes or sooner if new information becomes available. We appreciate your continued patience and understanding. Dstny Support

5. monitoring Oct 09, 2025, 12:51 PM UTC

   Microsoft has now confirmed that 96% of impacted resources have been successfully recovered as part of their ongoing mitigation efforts within Azure. Work continues to restore the remaining 4%, and we are closely monitoring progress. From our own testing, we can confirm that navigation within the Call2Teams portal is performing smoothly, and synchronisation is operating within expected timelines. Additionally, login to the Carrier Automate portal is working as expected. While services are showing strong signs of recovery, our engineering teams remain actively engaged and will continue to monitor for any residual impact. We will provide a further update in 60 minutes or sooner should new information become available. We appreciate your continued patience and understanding. Dstny Support

6. monitoring Oct 09, 2025, 02:10 PM UTC

   Microsoft has advised that Azure customers using Azure Front Door (AFD) should now be seeing consistent availability, with slightly elevated latencies continuing to improve as recovery progresses. However, some customers may still experience intermittent issues until Microsoft confirms full resolution of the incident. We are continuing to monitor the situation closely and will provide a further update in 60 minutes. Thank you for your continued patience and understanding. Dstny Support

7. monitoring Oct 09, 2025, 04:07 PM UTC

   Microsoft has advised that Azure customers using Azure Front Door (AFD) should now be seeing consistent availability, with slightly elevated latencies continuing to decrease as recovery progresses. Our engineering teams have observed no further issues within the Carrier Automate portal, the Call2Teams portal, or with Call2Teams synchronisation. All services continue to operate as expected. We will continue to monitor the situation closely and aim to provide a further update by 08:00 UTC tomorrow, or sooner if new information becomes available. Thank you for your continued patience and understanding. Dstny Support

8. resolved Oct 10, 2025, 08:03 AM UTC

   We are pleased to confirm that this incident has now been fully resolved. Microsoft has advised that, following continuous monitoring of latency improvements, they have declared the issue mitigated after receiving confirmation of recovery from affected customers. Over the past 16 hours, our engineering teams have also observed no further issues within the Carrier Automate portal, the Call2Teams portal, or with Call2Teams synchronisation. All services continue to operate as expected. The incident was caused by an issue originating with a third-party provider (Microsoft Azure). We are actively engaging with them to gain a full understanding of the root cause and to explore steps to prevent similar occurrences in future. Please note that Microsoft’s incident timelines differ from our own. Their documentation states that they will complete an internal retrospective within 3 days and aim to publish a Post Incident Review to impacted customers within 14 days. While we will make every effort to provide a detailed post-mortem, this will be on a best-endeavours basis and dependent on receiving the necessary information from Microsoft. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. Should you have any further questions or concerns, please don’t hesitate to contact our support team. Thank you, Dstny Support

9. postmortem Oct 14, 2025, 02:01 PM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Jonathan Goodier **Date Post Mortem Completed \(UTC\)** 14th Oct 2025, 13:30 **Incident Summary** Between 07:50 UTC and 16:00 UTC on 9th October 2025, Dstny services including Carrier Automate, Call2Teams, and Call2Teams Go experienced degraded performance and intermittent errors when accessing the service portals and completing account syncs. The issue was caused by a major service degradation in Microsoft Azure Front Door \(AFD\), which is used to route traffic to these services. The incident was proactively detected via monitoring alerts and escalated internally. Microsoft confirmed the issue as a multi-region capacity loss affecting over 200 AFD environments. Partial recovery began mid-morning, and full resolution was confirmed by the following day. **Root Cause** The incident was triggered by a latent bug in the Azure Front Door data plane. During onboarding of customer workloads to a newer control plane version, erroneous metadata was created due to a software defect. A cleanup operation of this metadata exposed the latent bug, causing infrastructure crashes across multiple edge sites in Europe and Africa. This led to increased latency and timeouts as traffic was redistributed to remaining healthy nodes, which became overloaded. Dstny services were impacted due to their reliance on Azure Front Door for routing, firewalling, and load balancing. This caused slow portal responses, sync failures, and intermittent loading issues for Call2Teams Go. **Incident Resolution** Microsoft restored Azure Front Door service through a combination of automated restarts and manual intervention. Failover operations were also initiated for critical services. Dstny monitoring confirmed that partial recovery from Microsoft mitigated most customer-facing impact by midday on 9th October. Full resolution was confirmed by 16:00 UTC on 9th October. No emergency changes were required internally, and service restoration was achieved without modification to Dstny infrastructure. **Mitigative Actions** To reduce the impact of similar incidents in future: * Microsoft has fixed the control plane defect and is actively resolving the latent data plane bug \(expected completion: October 2025\). * Microsoft is improving recovery time for data plane resource exhaustion \(expected completion: March 2026\). * Dstny is reviewing failover strategies and evaluating the use of Azure Traffic Manager to provide alternative routing paths during AFD outages. * Monitoring granularity will be enhanced to better isolate third-party infrastructure issues and improve incident response time. **Pending Vendor Report** Microsoft has published a preliminary report outlining the cause and resolution of the Azure Front Door outage. A final post-incident review is expected but has not yet been scheduled. Once available, it will be reviewed in full and any additional mitigative actions will be taken as required. The final report will be published at: [https://azure.status.microsoft/en-us/status/history](https://azure.status.microsoft/en-us/status/history)

Read the full incident report →

Started Oct 09, 2025, 06:45 AM UTC· resolved Oct 13, 2025, 08:32 AM UTC

1. investigating Oct 09, 2025, 06:45 AM UTC

   We are currently investigating this issue.

2. investigating Oct 09, 2025, 07:08 AM UTC

   We are continuing to investigate the ongoing issues with the ConnectMe application as a high priority. While the previous recommendation to uninstall and reinstall the app resolved some cases, we can confirm that it is not resolving all reported problems. Please do not attempt to uninstall and reinstall the application at this time. Our technical teams are actively working to identify the root cause and implement a comprehensive solution. We will provide the next update within 60 minutes. Thank you for your continued patience and understanding.

3. monitoring Oct 09, 2025, 07:47 AM UTC

   Our engineering team has identified the root cause of the service disruption and taken emergency action to restore services. Following this change, we are currently observing unsuccessful login attempts for ConnectMe desktop users. We kindly ask affected users to reattempt login. If issues persist, please contact the Dstny Support Team immediately for assistance. We are actively monitoring service availability and login activity. A further update will be provided in 2 hours. Thank you for your patience and understanding.

4. monitoring Oct 09, 2025, 10:49 AM UTC

   Our engineering team has identified the root cause of the service disruption and implemented corrective measures to restore services. Our Support team has not received any further reports of unsuccessful logins, and we have not observed any failed login attempts for ConnectMe desktop users. We will continue to monitor service availability over the next 24 hours and do not anticipate any further impact at this time. Thank you for your continued patience and understanding. Dstny Support

5. resolved Oct 13, 2025, 08:32 AM UTC

   Dear Customer, We are pleased to confirm that this incident has now been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. The root cause has been identified, and preventative measures have been implemented to avoid future occurrences. To ensure transparency and provide further insight, a detailed post-incident report will be made available within the next five working days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. If you have any further questions or concerns, please do not hesitate to contact our support team. Kind regards, Dstny Support

6. postmortem Oct 14, 2025, 02:00 PM UTC

   **Major Incident Category** Service Outage **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 14 Oct 2025, 11:15 **Incident Summary** Between 22:10 UTC on 8th October and 07:30 UTC on 9th October, ConnectMe was inaccessible for all users in the EU-West production environment, across both web and desktop platforms. The issue was introduced during a scheduled change initiated at 22:10 UTC on 8th October and was identified through routine post-upgrade testing. The outage was caused by the system continuing to use cached data containing outdated configuration, which prevented ConnectMe from refreshing correctly. Recovery efforts were coordinated promptly, and full service was restored by 07:30 UTC. **Root Cause** The outage was caused by a mismatch in how the application identified itself during login. A recent change had updated the way ConnectMe is recognised by the system, but the update wasn’t picked up by users’ browsers. This happened because the file that tells the browser how to load ConnectMe didn’t appear to have changed, even though it had. As a result, the browser continued using an older version of that file, which contained outdated information. This caused the system to reject login attempts and led to errors across the platform. **Incident Resolution** To restore service, the system was reconfigured to accept both the previous and updated client ID formats. This ensured compatibility and allowed users to log in successfully. Once this adjustment was made, normal functionality was restored across both web and desktop platforms. **Mitigative Actions** To help prevent similar issues in future, we are enhancing our testing processes across both web and desktop platforms, improving how changes are validated before release, and ensuring greater visibility of system behaviour following updates. We are also reviewing how changes are assessed for potential impact and strengthening coordination between teams during upgrades to ensure early detection and resolution of any unexpected behaviour. **Timeline \(UTC\)** * 08 Oct 2025, 22:00 - A system change was implemented in the EU-West production environment. * 09 Oct 2025, 00:47 - Upgrade completed. * 09 Oct 2025, 05:25 - Post-upgrade checks identified errors affecting ConnectMe. * 09 Oct 2025, 05:41 - Issue escalated to out-of-hours support. * 09 Oct 2025, 05:44 - Response teams engaged for coordinated investigation. * 09 Oct 2025, 06:09 - Customer impact confirmed by Dstny Support. * 09 Oct 2025, 06:12 - Major Incident process initiated. * 09 Oct 2025, 06:25 - WarRoom established to manage resolution efforts. * 09 Oct 2025, 06:30 - System adjustments made to restore service. * 09 Oct 2025, 07:30 - Successful logins confirmed; incident closed.

Read the full incident report →

Started Oct 01, 2025, 02:46 PM UTC· resolved Oct 02, 2025, 10:10 AM UTC

1. identified Oct 01, 2025, 02:46 PM UTC

   We are currently investigating a potential sync incident affecting Call2Teams across all regions. Our teams are actively working to identify the scope of the issue and we will provide updates every 60 minutes as we gather more information. Thank you for your patience as we work to address this matter. Dstny Support

2. identified Oct 01, 2025, 03:01 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Oct 01, 2025, 03:43 PM UTC

   We are currently investigating an issue affecting Call2Teams across all regions. This is causing sync timeouts for users globally. Our Customer Experience Engineering team has identified that accounts with a larger number of users are more likely to encounter this issue. They are actively working to implement measures to mitigate these sync timeouts. As part of the mitigation efforts, syncs were temporarily paused between 15:16 UTC and 15:23 UTC while engineering increased the timeout value to help partially alleviate the issue. Engineering teams are continuing to investigate the root cause, and a further update will be provided within the next 60 minutes. We apologise for any inconvenience caused and appreciate your patience during this time. Dstny Support

4. identified Oct 01, 2025, 04:47 PM UTC

   Following earlier mitigation efforts, increasing the timeout value has led to some improvement, with syncs now intermittently completing successfully. However, some failures are still occurring, and performance remains inconsistent. To further support stability, the timeout has now been increased to five minutes. Our engineering teams continue to investigate the underlying cause, focusing on both network and database-related factors. While these investigations are ongoing, customers may continue to experience slower-than-normal syncs and occasional interruptions until the root cause is identified and resolved. We remain committed to resolving this issue and will provide further updates from 08:00 UTC. We apologise for the inconvenience and thank you for your continued patience. Dstny Support

5. monitoring Oct 01, 2025, 08:39 PM UTC

   Our engineering teams have identified the root cause of the recent sync failures. A fix has been successfully implemented, and sync operations are now completing as expected. We will continue to monitor system performance closely as a precautionary measure.

6. resolved Oct 02, 2025, 10:10 AM UTC

   We are pleased to confirm that this incident has now been fully resolved. Over the past 12 hours, we have closely monitored the situation and observed no recurrence or further impact. The root cause has been identified, and preventative measures have been implemented to avoid future occurrences. For transparency and to provide further insight, a detailed post-mortem report will be made available within the next five business days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout this incident. If you have any further questions or concerns, please don’t hesitate to contact our support team. Kind regards, Dstny Support

7. postmortem Oct 07, 2025, 08:54 AM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 11th Oct 2025, 15:00 **Incident Summary** On 1st October 2025, between 08:30 and 18:40 UTC, some Call2Teams customers experienced issues with user syncs following a system update. This caused delays that led to sync failures, particularly for organisations with larger user volumes. While user changes could not be processed during this time, existing users remained able to make and receive calls. A temporary fix was applied to reduce the impact, and a permanent solution was deployed by 18:40 UTC. **Root Cause** The incident was triggered by a recent change to the Call2Teams API, which facilitates user synchronisation and acts as a bridge between customer telephony systems \(PBX or SIP trunks\) and Microsoft Teams. The update unintentionally affected sync reliability for organisations with large user volumes, leading to timeouts and preventing user updates from completing. **Incident Resolution** A temporary fix was applied to help more syncs complete despite slower system performance. Once the root cause was identified, a permanent solution was developed and deployed to restore normal service. **Mitigative Actions** To help prevent similar issues in future, we are enhancing our testing processes to better reflect real-world usage, including larger customer scenarios. We are also improving alerting to detect elevated sync failure rates earlier, and reviewing timeout handling to strengthen system resilience under load.

Read the full incident report →

Started Sep 23, 2025, 01:14 PM UTC· resolved Sep 23, 2025, 02:14 PM UTC

1. resolved Sep 24, 2025, 09:10 AM UTC

   Sync stopped working since 14:14 until 14:50 UTC 23rd September impacting all regions.

Read the full incident report →

Started Sep 11, 2025, 12:28 PM UTC· resolved Sep 12, 2025, 10:15 AM UTC

1. investigating Sep 11, 2025, 12:28 PM UTC

   We are currently investigating a potential sync related incident affecting Call2Teams. At this time, the specific regions and the extent of the impact are not yet confirmed. Our teams are actively working to identify the scope of the issue and we will provide updates every 60 minutes as we gather more information. Thank you for your patience as we work to address this matter. Dstny Support

2. investigating Sep 11, 2025, 01:11 PM UTC

   We can confirm that the sync functionality is failing for all users at this time. As a precaution, we kindly ask that partners refrain from attempting to perform sync operations until further notice. Please note that all other product functionality remains unaffected. We will provide a further update within the next 60 minutes. Thank you. Dstny Support

3. monitoring Sep 11, 2025, 01:20 PM UTC

   Our engineering team has implemented corrective measures to restore sync functionality. We can now confirm that account syncs are processing successfully across all production regions. Partners may resume account sync operations, including re-syncing any accounts that previously failed during the incident window. We will continue to monitor the Call2Teams services and sync activity over the next 24 hours, and will provide further updates if necessary. Thank you for your continued patience and support. Dstny Support

4. resolved Sep 12, 2025, 10:15 AM UTC

   We are pleased to confirm that this incident has now been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. The root cause has been identified, and preventative measures have been implemented to mitigate the risk of future occurrences. To ensure transparency and provide further insight, a detailed post-incident report will be made available within the next five working days. We sincerely apologise for any inconvenience caused and thank you for your patience and understanding throughout the duration of this incident. If you have any further questions or concerns, please do not hesitate to contact our support team. Thank you, Dstny Support

5. postmortem Sep 15, 2025, 04:36 PM UTC

   **Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 15th Sept 2025, 12:50 **Incident Summary** On 11th September, between 10:25 and 13:09 UTC, a service interruption affected all Teams sync processes on the Call2Teams platform. During this time, actions such as adding new users or updating user configurations could not be completed. Core calling functionality remained unaffected, allowing existing users to continue making and receiving calls. The Engineering team investigated and identified a recent networking change as the source of the issue. At 13:09 UTC, the change was reverted and redeployed, which restored normal service and allowed sync jobs to resume successfully. **Root Cause** The incident was caused by a change to the platform’s network settings, introduced as part of ongoing efforts to improve system security. This change unintentionally disrupted the platform’s ability to identify and reach the services it depends on, similar to removing entries from a phone book and being unable to make a call. As a result, sync operations failed, which affected user provisioning and configuration updates. The change was introduced as part of planned improvements to the platform and was not anticipated to affect live services. However, it resulted in a configuration that impacted the platform’s ability to connect with key services, leading to the disruption. We regret the impact this had and are reviewing our processes to help prevent similar occurrences in future. **Incident Resolution** The Engineering team investigated and confirmed the networking change as the source of the issue. At 13:09 UTC, the change was reverted and redeployed at 13:09 UTC, which successfully restored the platform’s ability to perform sync operations. Continuous monitoring confirmed that services were stable and functioning as expected following the recovery. **Mitigative Actions** To reduce the risk of similar incidents in future, we are implementing targeted improvements. A new alert will be introduced to proactively notify the team if no sync jobs succeed or fail within a defined period. Additionally, the planned DNS changes affecting the Platform Manager and Allocation Manager will be re-evaluated and restructured to ensure they are safely implemented with appropriate safeguards and testing. **Timeline \(UTC\)** * 11th Sept 2025, 10:25 - Sync failures began on the Call2Teams platform. * 11th Sept 2025, 10:28 - Initial alerts were received by the Engineering team. * 11th Sept 2025, 10:29 - First support ticket received from a partner. * 11th Sept 2025, 11:51 - Second support ticket received from a different partner, confirming wider impact. * 11th Sept 2025, 12:09 - Major Incident declared. * 11th Sept 2025, 12:17 - Response coordination initiated. * 11th Sept 2025, 12:26 - Engineering team engaged to investigate. * 11th Sept 2025, 12:50 - Networking change identified as the likely cause. * 11th Sept 2025, 13:09 - Change reverted, and service restored.

Read the full incident report →

Started Aug 21, 2025, 02:20 PM UTC· resolved Aug 22, 2025, 09:59 AM UTC

1. investigating Aug 21, 2025, 02:20 PM UTC

   We are currently investigating a potential incident affecting ConnectMe in the Nordics production region. Our teams are actively working to identify the scope of the issue and we will provide updates every 60 minutes as we gather more information. Thank you for your patience as we work to address this matter. Dstny Support

2. monitoring Aug 21, 2025, 02:51 PM UTC

   Our Engineering team have identified the root cause, and implemented actions to restore services to ConnectMe. Users will need to restart the ConnectMe application before attempting to log in. We continue to monitor service availability closely, and will provide a further update in the next 2 hours.

3. monitoring Aug 21, 2025, 03:32 PM UTC

   Our Platform team has identified the root cause of the issue and implemented corrective measures to restore application services. We will continue to monitor service availability for the next 24 hours. and do not anticipate any further impact at this time. Thank you. Dstny Support.

4. resolved Aug 22, 2025, 09:59 AM UTC

   We’re pleased to confirm that this incident has been fully resolved. Over the past 24 hours, we have closely monitored the platform and observed no recurrence or further impact. The root cause has been identified, and measures have been implemented to prevent a similar issue from occurring in future. To provide transparency and insight, a detailed post-incident report will be made available within the next five working days. We sincerely apologise for any inconvenience caused and appreciate your patience and understanding throughout this incident. If you have any further questions or concerns, please don’t hesitate to contact our support team. Kind regards, Dstny Support

5. postmortem Oct 16, 2025, 09:34 AM UTC

   **Major Incident Category** Service Outage **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 28 Aug 2025, 13:45 **Incident Summary** On 21st August 2025 at 13:56 UTC, a configuration error occurred during a routine maintenance activity involving the addition of a storage disk. This caused the storage platform to become overloaded, leading to system unresponsiveness and degraded performance. This led to a temporary outage for ConnectMe users in the Nordics region, while Analytics services experienced partial disruption, preventing data synchronisation. Recovery actions began at 14:30 UTC, including the removal of the newly added disk to stabilise the system. Full service was restored by 15:37 UTC, with no data loss reported. **Root Cause** The issue originated from an incorrect implementation during a standard infrastructure change to expand storage capacity. A recent update to the storage platform introduced undocumented changes to the disk integration process, requiring additional steps not present in previous versions. These changes were not known to the engineering team at the time. As a result, the disk addition triggered a network topology conflict between storage clusters. This caused unexpected system strain, leading to elevated CPU usage due to repeated timeouts and retries. The platform reached its operational limit, which in turn degraded performance and disrupted services. This disruption impacted service availability, resulting in a loss of service for all ConnectMe users in the Nordics region. Analytics services experienced a partial disruption, where users were unable to synchronise new data, although existing data remained accessible and the application itself remained stable throughout the incident. **Incident Resolution** To resolve the issue, a recovery plan was agreed at 14:30 UTC to remove the newly added disks from the storage platform. This action enabled the system to automatically rebalance and gradually return to normal operating levels. As a result, service performance stabilised, and full functionality was restored by 15:37 UTC. No data loss occurred during the incident. **Mitigative Actions** To reduce the risk of recurrence, several actions are underway: * Validation of new storage platform versions will be formally embedded into internal processes to ensure compatibility and stability before deployment. * A focused review of the Nordics storage expansion will help identify contributing factors and improve planning for future changes. * The process for adding storage disks has been reclassified from a standard to a normal change, allowing for greater oversight and risk management during infrastructure maintenance involving capacity adjustments. These steps are designed to strengthen platform resilience and ensure service continuity going forward. ### **Timeline**

Read the full incident report →

Started Aug 19, 2025, 12:38 PM UTC· resolved Aug 21, 2025, 02:55 PM UTC

1. investigating Aug 19, 2025, 12:38 PM UTC

   We are currently investigating an issue affecting Call2Teams across multiple regions. This is resulting in sync problems, causing users to be removed and appear as unattached within the Call2Teams portal. Consequently, affected users are experiencing a loss of registrations. Our teams are actively working to identify the root cause and implement a resolution. Updates will be provided every 60 minutes as more information becomes available. We sincerely apologise for any inconvenience caused and appreciate your patience during this time. Kind regards, Dstny Support

2. investigating Aug 19, 2025, 01:10 PM UTC

   We continue to investigate this incident in collaboration with our Platform team and are actively implementing measures to minimise user impact wherever possible. As part of the mitigation efforts, affected customers have been advised to perform another sync to restore user associations. Please note that users will only reattach to existing PBX components if there is a matching characteristic (such as email address or external ID). Where no match is found, users may need to be manually recreated. We will provide a further update within the next 60 minutes. Thank you for your continued patience. Kind regards, Dstny Support

3. investigating Aug 19, 2025, 02:41 PM UTC

   We continue to work closely with our Platform team to investigate this incident and minimise disruption to affected users. As part of our ongoing efforts, we are reviewing potentially impacted accounts and actively working with confirmed affected customers to restore data and ensure service continuity. We will provide a further update within the next 60 minutes. Thank you for your continued patience. Kind regards, Dstny Support

4. investigating Aug 19, 2025, 03:47 PM UTC

   Our Platform team has identified the root cause of the issue, and affected accounts now appear to be back online. No further action is required at this time. We are planning future improvements to prevent similar issues from occurring again and will continue to monitor service availability over the next 24 hours. We do not anticipate any further impact. Thank you for your continued patience. Kind regards, Dstny Support

5. monitoring Aug 20, 2025, 07:55 AM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Aug 20, 2025, 01:02 PM UTC

   Our Platform team continues to monitor service availability while investigating the issue. We’re planning improvements to prevent similar incidents in future. No further reports of impact have been received, and we do not anticipate any disruption to live services. Thank you for your continued patience. Kind regards, Dstny Support

7. monitoring Aug 21, 2025, 02:54 PM UTC

   We are continuing to monitor for any further issues.

8. monitoring Aug 21, 2025, 02:55 PM UTC

   We are pleased to confirm that this incident has been fully resolved. Over the past 24 hours, we have closely monitored the situation and observed no recurrence or further impact. We have determined the root cause and have implemented measures to prevent future occurrences. To provide transparency and insight into the incident, a detailed post-mortem report will be made available within the next 5 business days. We sincerely apologize for any inconvenience caused and thank you for your patience and understanding throughout this incident. Should you have any further questions or concerns, please feel free to reach out to our support team. Thank you, Dstny Support

9. resolved Aug 21, 2025, 02:55 PM UTC

   This incident has been resolved.

10. postmortem Oct 16, 2025, 09:38 AM UTC

    **Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 27 Aug 2025, 15:00 **Incident Summary** Between 16:30 UTC on 18 August and 13:10 UTC on 19 August 2025, a failure in the sync process logic led to the unintended deletion of users from the Call2Teams platform. During a period of high load, the system encountered a SQL error caused by exceeding the database’s connection limit. This error disrupted a critical safeguard designed to prevent user deletions, but the sync process continued running instead of halting. As a result, service users were erroneously removed from the database. Affected users became unattached and unlicensed, which prevented access to Call2Teams and Call2Teams Go services throughout the incident. The issue was identified through partner support tickets and confirmed by the Call2Teams Engineering team. A subsequent sync restored most users, while manually configured accounts required customer intervention to fully restore services. **Root Cause** The incident was caused by a failure in the system’s sync process during a period of high activity. The platform relies on Microsoft Graph to query user data, but a spike in invalid location responses triggered widespread error handling. This placed unexpected load on the database, which then failed to connect properly at a critical point in the sync. As a result, the system was unable to correctly flag users who should have been retained, and mistakenly removed them, leading to service disruption. **Incident Resolution** Once the issue was identified, our Engineering team recovered a backup of the user database to identify all affected accounts. A manual sync process was then used to restore services for impacted users. While most accounts were successfully recovered, some required partner intervention to complete the sync and fully restore access. **Mitigative Actions** To prevent recurrence, a proactive alert was introduced to detect similar issues early and enable faster response. The affected database was manually scaled to improve performance under high load. Updates to retry logic are scheduled for release, and improvements to safeguard logic are currently being planned, with implementation date to be confirmed. ### **Timeline \(UTC\)** * 18 Aug 2025, 16:30 - Incident begins; elevated database load observed. * 18 Aug 2025, 17:30 - Database load returns to normal naturally. * 19 Aug 2025, 08:30 - Support query raised; initial investigation initiated by CX Engineering. * 19 Aug 2025, 09:30 - Re-syncing identified as a potential recovery method. * 19 Aug 2025, 12:17 - Major Incident raised; partner communications issued via Zone shortly after to inform stakeholders. * 19 Aug 2025, 13:00 - Backup from 18th Aug used to assist with affected accounts. * 19 Aug 2025, 13:10 - Mitigation identified; customers advised to perform sync to restore users. * 19 Aug 2025, 13:45 - Incident moved to monitoring; platform improvements initiated. * 20 Aug 2025, 16:00 - Root cause analysis identifies database load triggered by exception handling. * 21 Aug 2025, 15:30 - Error handling confirmed; database scaled to reduce future risk.

Read the full incident report →