Degraded Performance of Phrase Orchestrator (EU) Next-Gen Workflow Engine between July 28, 05:15 AM CEST and July 28, 09:56 AM CEST

Update timeline

1. investigating Jul 28, 2026, 07:19 AM UTC

   Investigating - Engineering has identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.

2. investigating Jul 28, 2026, 07:53 AM UTC

   We are continuing to investigate this issue.

3. identified Jul 28, 2026, 08:30 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Jul 28, 2026, 08:35 AM UTC

   A fix has been implemented and we are monitoring the results. The workflow engine is processing the queue of pending executions.

5. resolved Jul 28, 2026, 11:49 AM UTC

   All executions have been processed. This incident has been resolved.

6. postmortem Jul 30, 2026, 07:09 AM UTC

   ## Introduction We would like to share details about an incident that affected Phrase Orchestrator on July 27–28, 2026. During this period, workflow executions in the Next-Gen Workflow Engine were unable to progress and remained stuck in an "executing" state. No data was lost during the incident. This post-mortem explains what happened, when it was resolved, and the steps we have taken to prevent a recurrence. ## Timeline * **27 July 2026 at 18:55 CEST** – The workflow engine began producing errors as database query performance degraded. Workflow executions stalled and stopped progressing. * **27 July 2026 at 20:54 CEST** – The first customer report of executions stuck in "executing" was received. * **27 July 2026 at 22:39 CEST** – The incident was formally declared. * **28 July 2026 at 02:18 CEST** – A service restart provided temporary relief; workflow executions resumed. * **28 July 2026 at 05:15 CEST** – The issue recurred as the underlying database performance problem persisted. * **28 July 2026 at 09:56 CEST** – The root cause was identified and addressed. No executions were lost; however, due to partial service restarts, some actions within executions were retried, which may have caused a small number of executions to fail that otherwise would have succeeded. * **28 July 2026 at 13:46 CEST** – The full backlog of stalled executions was confirmed as cleared. The system was declared stable. * **28 July 2026 at 13:48 CEST** – Incident resolved. ## Root Cause The incident was caused by progressive bloat in database indexes used by the workflow job scheduling system. The performance of these particular indexes gradually degraded over time as they accumulated dead index entries from prior writes and updates. The job scheduling engine acquires database-level coordination locks while querying these indexes to determine which jobs to dispatch. As the index lookups grew slower, they began exceeding the database's configured statement timeout. When a lookup was canceled by the timeout, the scheduling process responsible for that work crashed and restarted. With no schedulers running, no workflow steps could be dispatched and all in-progress workflow executions became stuck. The database server itself remained healthy throughout the incident, with normal CPU and connection levels. The problem was exclusively lock and latency contention within the scheduling layer. A service restart cleared the crashed processes and temporarily restored execution. However, because the index bloat was still present, the same degradation recurred once query load resumed. A manual index rebuild fully restored performance and resolved the issue. ## Actions to Prevent Recurrence 1. **Automated index maintenance added** – Scheduled automatic index maintenance has been configured for the affected indexes. This ensures bloat cannot accumulate over time and eliminates the conditions that triggered this incident. 2. **Legacy indexes removed** – Unused legacy database indexes have been identified and removed, reducing the overall maintenance surface and simplifying future index hygiene. 3. **Monitoring coverage updated** – Our monitoring landscape is being reviewed and updated to reflect the current state of the workflow engine. This work will close gaps that allowed the degradation to go undetected before the first customer report.