DeepL is experiencing a problem with Document Translations

Update timeline

1. identified Sep 09, 2026, 03:45 PM UTC

   We've identified an issue with our Document Translation service leading to a full service outage for the majority of our DocTras customers. We are investigating and will post an update shortly.

2. investigating Sep 09, 2026, 04:15 PM UTC

   We've traced the fault to a routine update in our storage layer. We are investigating possible resolution options and will update shortly.

3. resolved Sep 09, 2026, 04:55 PM UTC

   All systems are now functions correctly and service is fully restored. Apologies for the inconvienence. We will post an RCA document shortly.

4. postmortem Sep 16, 2026, 03:10 PM UTC

   Summary On 9 September 2026, between 16:44 and 17:46 UTC, customers using DeepL Pro document translation from our EU datacenter could not translate documents. Uploads in the web app failed, the document translation API (/v2/document) returned HTTP 500 error code, and an asynchronous translation endpoint used by a CAT-tool integration returned errors. Text translation, the free tier, and customers served from our other data centres were not affected. We are sorry for the disruption. It happened during a planned upgrade of the storage cluster behind these services, and the safeguards we relied on did not prevent it. This was not a security incident: no customer content was accessed, lost, or corrupted. Impact Customers hosted in the EU data centre Document translation, web and API: new document translations could not be started for about one hour. At the peak roughly four in five upload attempts were rejected. Checking on or downloading already-translated documents kept working. Pro API, asynchronous translation endpoint used by a CAT-tool integration: about 1.25 % of all Pro API calls failed in the first 30 minutes, almost all on this endpoint. Data: failed uploads were rejected before completion and never stored. Stored documents were untouched. Affected customers needed to retry their uploads once service was restored. Not affected: text translation and every other API endpoint, the free tier, all other data centres. What happened Document translation stores uploaded documents in an S3-compatible object store, built on the open-source Ceph storage system and operated with the open-source Rook orchestrator. Two kinds of component matter: gateways, which speak S3 to our applications, and storage servers, which hold the data. Larger documents are uploaded in parts and finalised with a "complete" request. Trigger. During the rolling upgrade from Ceph 19.2 to 20.2, the orchestrator upgraded all gateways as soon as the cluster's monitors were on the new version, then began the much longer, server-by-server storage upgrade. For about an hour, new gateways wrote to old storage servers — the reverse of Ceph's documented order. The orchestrator did not enforce that order, and our procedure relied on it. Cause. The new gateway version records a checksum for each uploaded part and requires all checksums to be present when the upload completes. Each part's record is written by the storage server holding it; an old storage server didn't know the new checksum field and silently dropped it. The gateway then found checksums missing and rejected completion with HTTP 400, even though every part had been stored correctly. Whether an upload failed depended only on whether its metadata's server had been upgraded yet — which is why failures disappeared as servers were upgraded. Recovery came from the upgrade progressing past the affected storage servers, not from a mitigation we applied. Alerts fired within three minutes of the first customer-facing error; the status page was updated 21 minutes after impact began, which we want to shorten. Why it was not caught earlier. Our services use the AWS SDK, which requests upload checksums by default. The previous storage version silently ignored them, so this path had never been tested against a server that enforces it. The same upgrade had passed on a smaller pre-production cluster, where the mixed-version window lasted only minutes because that cluster is much smaller, so the mismatch never had time to surface and the upgrade path was confirmed valid for production. Our storage monitoring captured availability and server-side errors, both of which stayed healthy, but the rejections were client-side errors and were not surfaced. Preventative Measures Completed Storage dashboards now show client-facing error rates per bucket and client library. This will bring incident detection for this failure mode within a minute. Both underlying defects are reported to the Ceph and Rook OSS projects maintainers. The Rook community merged a fix that correctly orders storage servers upgrade before gateways. In progress Once a new Rook version is available with the above mentioned fix, upgrade the Rook operator. Meanwhile pause Ceph storage upgrades where possible. Alternatively, if storage upgrades are required, manually keep the gateway version fixed. Add an alert for gateway client-error rates Make the client-side checksum behaviour a configuration setting, so that it’s fully visible application side and be able to modify it without requiring a code change. Planned Treat production storage upgrades as visible change windows, announced in the incident channel in addition to the current announcements to increase internal visibility, and marked on the affected dashboards. Document and test a rollback procedure for the gateway tier before the next version upgrade takes place.