Dixa Outage History

## **Summary** On April 8, 2026, Dixa experienced a partial outage lasting approximately 58 minutes. New WebSocket connections were unable to be established, which meant that new logins failed, and any agents who refreshed their browser or lost their connection could not reconnect. Agents who remained on an existing session were unaffected during the incident. The root cause was a TLS certificate misconfiguration introduced during a planned migration of our ingress controller infrastructure. The issue was identified, fixed, and fully resolved within the hour. ## **Impact** Between 18:26 and 19:24 CEST, customers attempting to log in to Dixa or re-establish a WebSocket connection \(e.g., after a page reload\) were unable to do so. Browsers rejected the connection due to an invalid TLS certificate being served. Agents who were already logged in with an active WebSocket session continued to operate normally throughout the incident. The impact was limited to new or reconnecting sessions. A small number of customers were affected and reported the issue to our support team. No conversations or data have been lost during the incident. ## **Timeline \(CEST\)** * 18:26 - Internal reports that Dixa is not loading for some users * 18:30 - Issue escalated to engineering via our critical support channel * 18:32 - Status page updated to **Investigating** * 18:40 - Engineering identifies a TLS certificate error \(`ERR_CERT_AUTHORITY_INVALID`\) * 18:52 - Root cause identified, a self-signed default certificate was being served instead of the correct one * 19:00 - Status page updated to **Identified** * 19:11 - Fix deployed; WebSocket connections begin recovering * 19:12 - Status page updated to **Monitoring** * 19:24 - Full recovery confirmed; status page updated to **Resolved** ## **Root Cause** As part of our ongoing WebSocket resilience work to ensure a more stable platform, we migrated our ingress controller \(the component that routes incoming traffic to internal services\) from an end-of-support solution to a new one. This migration was tested in our staging environment before being applied to production. However, there was a configuration discrepancy between staging and production for the ingress class that handles WebSocket traffic. When DNS was switched to the new ingress controller on the morning of April 8, existing connections continued to work through cached DNS entries still pointing to the old controller. Hours later, as DNS caches expired across the internet, clients began resolving to the new controller, which, due to the misconfiguration, did not recognize the WebSocket routes. This caused it to serve a default self-signed TLS certificate instead of the valid one, leading browsers to reject the connection. The length of the partial outage per customer varies depending on when the DNS cache expired, and if WebSocket connections started to connect to the new load balancer. ## **Resolution** Once the root cause was identified, a configuration update was deployed to the new ingress controller so that it could correctly handle WebSocket traffic. Connections began recovering immediately after the fix was applied. ## **Preventive Measures** We have taken the following steps to reduce the likelihood and impact of similar issues in the future: * **Environment parity:** All non-production environments have been aligned with production configuration conventions, eliminating the discrepancy that caused this incident. * **Endpoint monitoring:** We have added external monitoring checks that validate both the availability and TLS certificate validity of our WebSocket endpoints. This will enable faster detection if a similar issue occurs. * **WebSocket isolation**: We will isolate the platform's WebSocket requirement to be optional, so if a similar issue should happen in the future, the disruption will be less intrusive for users. ‌ ‌ We sincerely apologize for the disruption this caused. Reliability is a top priority for us, and we are committed to learning from every incident to make Dixa more resilient. If you have any questions, please don't hesitate to reach out to your account team or our support at [[email protected]](mailto:[email protected]).

_Dixa experienced issues with the Smart Reply feature in AI CoPilot on 18 March 2026._ _**Summary**_ _18 March 2026 at 14:37 CET - 20:20 CET, customers experienced issues with the Smart Reply feature in AI CoPilot. Agents were unable to use Smart Reply._ _**Root cause**_ _A code change introduced a regression that caused agents' browsers to block completing the request to load Smart Reply suggestions._ _**Timeline**_ _At 14:37 CET on 18 March 2026: Issue flagged internally after several customers reported the issue_ _At 15:33 CET: Engineering investigates and attempts an initial rollback, which does not resolve the issue._ _At 16:02 CET: Status page updated to "Investigating" — customers notified of difficulties with AiCoPilot Smart Replies._ _At 16:48 CET: Status page updated — engineering continues to investigate._ _At 18:17 CET: Status page updated — root cause investigation ongoing_ _At 20:20 CET: The knowledge-backend service is successfully rolled back to a stable version, restoring Smart Reply._ _At 20:52 CET: Status page updated to "Resolved" — all known issues confirmed resolved._ _**Solution**_ _The immediate solution was to roll back to a stable version from prior to the breaking change, restoring Smart Reply for all affected customers._ _Long term, the team will review the offending commit before re-deploying to prevent similar regressions from reaching production undetected._ _We sincerely apologise for the inconvenience caused by this issue._

**Summary:** On March 11, 2026, Dixa experienced platform-wide degraded performance lasting approximately 3 hours \(08:30 - 12:35 CET\). Customers experienced slow or failed conversation loading, timeouts on email sending, conversation transfers, assignments, and flow processing. No data was lost, and there were no security issues at any point. **Impact:** * _Availability_: Platform-wide slowness and partial inaccessibility for ~3 hours. * _Affected functionality_: Conversation loading, email sending, conversation transfers, conversation assignments, and flow processing - all experienced significant slowness and intermittent failures. * _Data integrity:_ All emails were fully processed after the fix. No data was lost, and no security issues occurred at any point. **Root Cause:** The incident was caused by an atypical traffic pattern in inbound email processing that resulted in repeated internal retries - retries are a normal part of email distribution, accounting for factors such as sending delays and server availability, but this expanded exponentially. The sustained retry volume placed excessive load on a central platform component, causing cascading timeouts across dependent services and resulting in platform-wide degradation. **Timeline \(CET\):** * Mar 11, 06:00 - First signs of email processing errors detected * Mar 11, 08:30 - Platform degradation begins; customer impact starts * Mar 11, 10:50 - First mitigation deployed; partial improvement * Mar 11, 12:30 - Root cause fully identified; final fix applied * Mar 11, 12:35 - Platform stability confirmed **Resolution:** We identified and addressed the source of the abnormal email volume, which resulted in an immediate reduction in error rates, and the platform to recover. **What We Have Done Since This Incident:** Following this incident, we have already implemented the following improvements: 1. Added validation to reject invalid email addresses early in the pipeline, preventing them from entering retry loops. 2. Optimised internal lookups to fetch only necessary data instead of the full conversation history, significantly reducing load during email processing. 3. Added deduplication logic to prevent redundant data fetches during email processing. 4. Enforced concurrency limits: platform components now shed excess traffic when saturated, allowing requests to be redistributed rather than queued indefinitely. 5. Added deadline checking: expired requests are now discarded immediately instead of consuming resources on work that is no longer needed. 6. Reduced internal timeout thresholds to fail fast under contention rather than blocking for extended periods. **What We're Continuing to Work On:** 1. Loop detection and interruption - Introduce mechanisms to detect and automatically halt email processing anomalies before they can accumulate significant load. 2. Improved alerting and escalation - Ensure processing anomalies are detected and escalated with appropriate urgency. **Closing Note:** We sincerely apologize for the disruption this caused. These improvements are our highest priority. If you have any questions, please reach out to [[email protected]](mailto:[email protected]).