Dstny Outage History

**Incident Summary** From 13:45 UTC on 13th April 2026 until 11:29 UTC on 15th April 2026, a subset of Call2Teams customers experienced intermittent call quality issues. Reported symptoms included one‑way audio, silence at the start of calls or during internal transfers, intermittent static, and in some cases early call termination shortly after connection. While initial reports suggested a possible regional limitation, investigation confirmed that affected scenarios could occur across all regions. Following a series of targeted fixes and monitoring, full service was restored and confirmed stable by 11:29 UTC on 15th April 2026. **Root Cause** The incident was caused by an interaction between a change in Microsoft’s call setup behaviour and a previously latent issue in the Call2Teams call setup logic. Microsoft’s updated codec selection exposed a defect in how our platform handled codec ordering in certain responses, which could create mismatches between the codec signalled to customer phone systems and the codec used towards Microsoft. In parallel, G.722 was being advertised to Microsoft in line with SBC vendor guidance and earlier interoperability assumptions, leading Microsoft to select G.722 in cases where some customer PBXs did not support it. Together, these factors increased real‑time transcoding and contributed to the intermittent audio problems observed. **Incident Resolution** Engineers first reduced unnecessary transcoding by adjusting configuration so that G.722 is only offered to Microsoft when it is explicitly supported by the customer’s phone system. Outside of the standard default codecs \(PCMA and PCMU\), advertised codecs were aligned with each customer’s PBX capabilities, which immediately improved platform stability. A software update was then deployed to correct call setup handling so that codec selections negotiated with Microsoft are maintained consistently when signalled back to customer systems. Following rollout, targeted testing, and monitoring, call quality was confirmed to be stable across all affected scenarios. **Mitigative Actions** We have corrected the call setup logic to prevent incorrect codec re‑ordering from being signalled to customer systems, ensuring that codec selections negotiated with Microsoft are preserved end‑to‑end. We have also reduced unnecessary transcoding by limiting advertised codecs to standard defaults \(PCMA and PCMU\), while conditionally including other codecs, such as G.722, only where they are explicitly supported by the customer’s phone system. In addition, we are reviewing our SBC configuration and internal processes to improve early detection of, and response to, third‑party behavioural changes, strengthening overall service resilience and reducing the likelihood of similar incidents recurring. ### **Timeline**

**Incident Summary** On 2nd April 2026, between 04:05 UTC and 07:37 UTC, a routine software update in the EU West region led to intermittent call failures for a subset of ConnectMe users. The issue was proactively detected, and service was fully restored after the update was rolled back at 07:37 UTC. All systems were subsequently verified as stable. **Root Cause** The incident was triggered by a routine software update aimed at enhancing service recovery within the ConnectMe platform. During deployment, the update caused an unexpected interaction within the system, leading to a key component becoming unresponsive and resulting in intermittent outbound call failures for some users. **Incident Resolution** The issue was resolved by rolling back the software update to the previous stable version at 07:37 UTC. This action restored normal service for all users. The rollback was performed promptly after the root cause was identified, and service stability was closely monitored to confirm full recovery. **Mitigative Actions** We are refining monitoring to better detect outbound call failures and reviewing the upgrade process to address identified gaps. Backend adjustments will also be made to improve stability in future deployments.

**Incident Summary** On 3rd March 2026, a subset of Call2Teams users in the US region experienced disruption to their calling services. Between 12:55 UTC and 16:50 UTC, users served by one of our Edge signalling SBCs were unable to register successfully, which prevented them from making or receiving calls during this period. Full service was restored for all affected users by 16:50 UTC. **Root Cause** The root cause was introduced during routine maintenance on one of our Edge signalling SBCs. Following a software upgrade, essential services were not restarted as required, leaving the SBC unable to process user registrations. This resulted in affected users being unable to establish calls during the disruption window. **Incident Resolution** Initial recovery actions began at 16:22 UTC, when key services on the affected Edge signalling SBC were restarted, restoring functionality for the majority of impacted users. A small number of users required an additional corrective action, and full service was restored for them at 16:50 UTC by temporarily redirecting their traffic to an alternative Edge to ensure continuity. The following day, all relevant services were comprehensively restarted to confirm stable and correct operation across the platform. **Mitigative Actions** * Strengthening maintenance procedures and post‑upgrade validation to prevent incomplete steps during routine operations. * Implementing enhanced monitoring to alert teams immediately if registration success rates drop to zero on any Edge.

**Major Incident Category** Service Degradation **Post Mortem Owner** Ant Hurlock **Date Post Mortem Completed \(UTC\)** 04 Mar 2026, 17:30 **Incident Summary** On 26th February 2026, a small percentage of ConnectMe users in the EU West region experienced service degradation between 08:19 and 09:22 UTC. Affected users were connected to a single platform node and encountered varied symptoms, including blank screens after login and disruptions to calling and related functionality. Although the overall number of impacted users was limited, the disruption for those affected was significant. Engineers applied mitigation steps, and all services were fully restored and confirmed stable by 09:22 UTC. **Root Cause** The incident was triggered by a software fault within a core platform component. A planned update containing improvements and fixes for two related issues had not yet been deployed, meaning these enhancements were not available in the production environment. The first issue caused the affected platform node to become unresponsive, and the diagnostic information required to quickly identify the cause was insufficient. The upcoming update will introduce enhanced logging to provide the visibility needed to detect and analyse node deadlock scenarios in real time. The second issue involved the node’s health‑monitoring probe. A related software defect meant the probe could take significantly longer than expected to detect and recover from failures, extending the automated recovery window to as much as two hours. Although the platform is designed to self‑recover within this timeframe, in this case recovery did not occur until engineers intervened manually to restore service. **Incident Resolution** The issue was first detected by automated monitoring at 08:19 UTC, prompting immediate investigation by our engineering team. Initial recovery efforts were delayed due to an incorrect procedure, which terminated the unhealthy pod but left the health‑check waiting on internal conditions that were not relevant to this type of failure. As a result, the health‑check began a recovery cycle that could have taken up to two hours to complete. Engineers then intervened manually, restarting the affected component and restoring normal service for all impacted users at 09:22 UTC. Service stability was confirmed following this intervention. **Mitigative Actions** * Accelerating deployment of the planned software update. * Improving the speed and reliability of automatic recovery by resolving the issue that previously caused the health‑monitoring probe to delay self‑recovery by up to two hours. * Introducing enhanced logging to support faster diagnosis of node deadlock scenarios and progress towards a permanent fix. * Updating runbooks to provide clearer guidance for rapid recovery and validation of service restoration.

**Incident Summary** On 14th February 2026, a critical outage affected all users in the Nordics production environment. The authentication service became unavailable, resulting in a complete loss of access to all authentication‑dependent services including ConnectMe, SMP, and others. The incident began at 00:27 UTC and was resolved for end-users by 05:15 UTC. **Root Cause** The outage was caused by an unintended configuration that allowed a point‑in‑time recovery backup file to grow unexpectedly. This rapid growth consumed the storage allocated to the authentication database. Once the storage was exhausted, the database crashed, rendering the authentication service \(Keycloak\) unavailable. As a result, all dependent services relying on authentication were unable to process login operations, leading to a widespread service outage. **Incident Resolution** Service was restored by increasing the storage capacity for the authentication database and identifying that an automated cleanup process was rapidly consuming the new space. The authentication service was temporarily scaled down to halt the excessive data growth, and an index was added to improve the cleanup process. Once these steps were completed, the authentication service was brought back online and user access was restored at 05:15 UTC. Work to restore full database redundancy continued after user impact ended. **Mitigative Actions** * The team is reviewing whether point-in-time recovery should be implemented for the authentication database. * The recovery procedure for the database is being overhauled and comprehensively updated to ensure a faster and more effective response in future incidents. * Steps are being taken to confirm that the bastion server is fully isolated and dedicated to production, ensuring separation from development environments. * Ongoing investigation into periodic CPU and network spikes related to Keycloak database queries is underway.