Issues with multiple DeepL services

Update timeline

1. investigating Aug 27, 2026, 02:45 AM UTC

   DeepL continues to investigate a problem our central authentication store that is affecting multiple services, including Accounts/Login & voice services.

2. investigating Aug 27, 2026, 04:20 AM UTC

   We continue to see performance issues with our central authentication system which is affecting the usage of a platform across a range of services. A number of successful measures were put in place to help stabalise the system to improve performance but we are still short of normal operating conditions and continue to work on a long term solution.

3. investigating Aug 27, 2026, 06:42 AM UTC

   We've identified the root cause of the sustained higher load and have out in temporary measures to manage the load and restore service to normal levels. We will continue to work on this until we have full restored serviced and resolved the underlying cause.

4. investigating Aug 27, 2026, 07:30 AM UTC

   Service is now largery restored for customers but there are delays in processing license changes. We will continue to work on resolvng the issue.

5. resolved Aug 27, 2026, 09:55 AM UTC

   Service has now been restored and license processing is up to date. We will continue to monitor any issues that arise.

6. postmortem Sep 01, 2026, 08:24 AM UTC

   Root Cause Analysis — Authentication Outage, 26–27 August 2026 Author: Richard Marshall, Head of Site Reliability Date: 28th August 2026 DeepL Incident ID: PIN-805 Summary Between 15:56 UTC on 26 August and ~07:00 UTC on 27 August 2026, sign-in and account operations on DeepL were degraded. A subscription change for a customer triggered a routine license update which, due to an account-linking fault, was applied to a far larger set of accounts than intended. The update couldn't complete, and our systems reprocessed it continuously instead of setting it aside, degrading shared infrastructure the sign-in path depends on. We are treating this as a Severity 1 incident — our highest — and apologise for the disruption. Impact Sign-in error rates exceeded the 5% unavailability threshold, peaking at ~40%; an external monitor recorded our login backend as down. Account info/entitlement retrieval was significantly impaired. Termbases, document translation, voice, and license management were degraded for parts of the window. Because the incident ran overnight in European hours, APAC business hours were most affected, and impact was uneven across services. Timeline (UTC) Time Event 26 Aug 15:56 Subscription change triggers license update against a much larger group than intended; sign-in errors begin 26 Aug 16:08 Incident opened 26 Aug eve–27 Aug early Infrastructure-layer mitigations reduce symptoms but don't resolve the incident 27 Aug ~06:50 License-processing service identified as source of sustained load; processing paused 27 Aug ~07:00 Load normal; service restored 27 Aug ~09:30 Unprocessable request removed, backlog cleared; incident fully mitigated 28 Aug Configuration corrected; incident review held What happened A subscription change for one customer triggered a routine license update. Due to an account-linking issue, the update targeted a much larger account group than intended: instead of a small administrative group, it linked to the customer's default directory-synchronised group. Why it caused an outage The request's size wasn't itself the problem — processed once, it would've been a brief, short-lived demand within normal limits, invisible to customers. The outage happened because the request couldn't complete successfully, and the system reprocessed it repeatedly rather than setting it aside. Each retry repeated the same large demand; the accumulated, sustained load — not a single spike — degraded shared infrastructure and caused sign-in failures. Two design characteristics turned one failed request into a sustained one: the work was issued as one large operation under a single time limit, so it couldn't finish in time; and when it didn't finish, it went back to the front of the queue and retried immediately, with no attempt limit and nowhere else to go. Why it took time to resolve Early symptoms resembled a pattern seen before in our database layer — a reasonable read at the time — so initial remediation focused there, reducing symptoms but not addressing the underlying cause, identified only the next morning. The key fix — pausing license processing — wasn't found until a fresh team took over the next morning and, seeing the growing queue sizes, questioned how the queued messages were interacting with the wider system. Resolution Once license processing was paused, normal service resumed. The unprocessable request was identified, removed, and affected groups were temporarily excluded from automatic license processing to prevent recurrence, letting the backlog clear. The account-linking error has since been corrected and the temporary exclusions removed. A small number of protective database-layer settings applied during the incident remain in place deliberately, pending confirmation of sustained normal performance. What we are doing about it Agreed at the 28 August incident review: Preventing the same trigger Break large license updates into independently processed, queued batches instead of one operation under a single time limit. Already in progress. Remove the account-linking fallback: subscriptions will no longer fall back to a default group; the target group must be specified explicitly. Ensuring a single bad request can't cause an outage Route unprocessable requests to a holding queue after a limited number of attempts instead of retrying indefinitely — isolated for investigation, reprocessed deliberately. Being implemented for the affected service and adopted as a broader messaging-infrastructure standard. Detecting and resolving incidents faster Service run books: DeepL's been rolling these out for call-outs, but this service lacked one. To be corrected, with the owning team defining it in coming weeks.