Is Phrase down?

We are currently investigating an issue with branching events missing in Phrase Strings (EU), which may cause changes to not be applied when branches are merged.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented. The backlog of missing events is now being processed. Once the queue is fully processed, unmerged branches will automatically receive the missing changes. We are monitoring the queue and results.

Changes made to branches during the incident window are now fully applied. Merged branches that were affected have been identified and impacted customers have been contacted directly. This incident has been resolved.

We are currently investigating an issue affecting Connectors in Phrase TMS. Customers may be unable to access or use Connector functionality during this time. Our engineering team is actively working to identify the cause and restore full service as quickly as possible.

We are continuing to investigate this issue.

We have identified the root cause and the issue has now been resolved. Connectors should be appearing as active again. We will continue to monitor the situation to ensure full stability.

We continue monitoring the situation.

The incident has been resolved and all components are back to operational.

Investigating - Engineering has identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results. The workflow engine is processing the queue of pending executions.

All executions have been processed. This incident has been resolved.

## Introduction We would like to share details about an incident that affected Phrase Orchestrator on July 27–28, 2026. During this period, workflow executions in the Next-Gen Workflow Engine were unable to progress and remained stuck in an "executing" state. No data was lost during the incident. This post-mortem explains what happened, when it was resolved, and the steps we have taken to prevent a recurrence. ## Timeline * **27 July 2026 at 18:55 CEST** – The workflow engine began producing errors as database query performance degraded. Workflow executions stalled and stopped progressing. * **27 July 2026 at 20:54 CEST** – The first customer report of executions stuck in "executing" was received. * **27 July 2026 at 22:39 CEST** – The incident was formally declared. * **28 July 2026 at 02:18 CEST** – A service restart provided temporary relief; workflow executions resumed. * **28 July 2026 at 05:15 CEST** – The issue recurred as the underlying database performance problem persisted. * **28 July 2026 at 09:56 CEST** – The root cause was identified and addressed. No executions were lost; however, due to partial service restarts, some actions within executions were retried, which may have caused a small number of executions to fail that otherwise would have succeeded. * **28 July 2026 at 13:46 CEST** – The full backlog of stalled executions was confirmed as cleared. The system was declared stable. * **28 July 2026 at 13:48 CEST** – Incident resolved. ## Root Cause The incident was caused by progressive bloat in database indexes used by the workflow job scheduling system. The performance of these particular indexes gradually degraded over time as they accumulated dead index entries from prior writes and updates. The job scheduling engine acquires database-level coordination locks while querying these indexes to determine which jobs to dispatch. As the index lookups grew slower, they began exceeding the database's configured statement timeout. When a lookup was canceled by the timeout, the scheduling process responsible for that work crashed and restarted. With no schedulers running, no workflow steps could be dispatched and all in-progress workflow executions became stuck. The database server itself remained healthy throughout the incident, with normal CPU and connection levels. The problem was exclusively lock and latency contention within the scheduling layer. A service restart cleared the crashed processes and temporarily restored execution. However, because the index bloat was still present, the same degradation recurred once query load resumed. A manual index rebuild fully restored performance and resolved the issue. ## Actions to Prevent Recurrence 1. **Automated index maintenance added** – Scheduled automatic index maintenance has been configured for the affected indexes. This ensures bloat cannot accumulate over time and eliminates the conditions that triggered this incident. 2. **Legacy indexes removed** – Unused legacy database indexes have been identified and removed, reducing the overall maintenance surface and simplifying future index hygiene. 3. **Monitoring coverage updated** – Our monitoring landscape is being reviewed and updated to reflect the current state of the workflow engine. This work will close gaps that allowed the degradation to go undetected before the first customer report.

Engineering has identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.

All workflows are being executed again as expected.

## Introduction We would like to share details about an incident that affected Phrase Orchestrator on July 27–28, 2026. During this period, workflow executions in the Next-Gen Workflow Engine were unable to progress and remained stuck in an "executing" state. No data was lost during the incident. This post-mortem explains what happened, when it was resolved, and the steps we have taken to prevent a recurrence. ## Timeline * **27 July 2026 at 18:55 CEST** – The workflow engine began producing errors as database query performance degraded. Workflow executions stalled and stopped progressing. * **27 July 2026 at 20:54 CEST** – The first customer report of executions stuck in "executing" was received. * **27 July 2026 at 22:39 CEST** – The incident was formally declared. * **28 July 2026 at 02:18 CEST** – A service restart provided temporary relief; workflow executions resumed. * **28 July 2026 at 05:15 CEST** – The issue recurred as the underlying database performance problem persisted. * **28 July 2026 at 09:56 CEST** – The root cause was identified and addressed. No executions were lost; however, due to partial service restarts, some actions within executions were retried, which may have caused a small number of executions to fail that otherwise would have succeeded. * **28 July 2026 at 13:46 CEST** – The full backlog of stalled executions was confirmed as cleared. The system was declared stable. * **28 July 2026 at 13:48 CEST** – Incident resolved. ## Root Cause The incident was caused by progressive bloat in database indexes used by the workflow job scheduling system. The performance of these particular indexes gradually degraded over time as they accumulated dead index entries from prior writes and updates. The job scheduling engine acquires database-level coordination locks while querying these indexes to determine which jobs to dispatch. As the index lookups grew slower, they began exceeding the database's configured statement timeout. When a lookup was canceled by the timeout, the scheduling process responsible for that work crashed and restarted. With no schedulers running, no workflow steps could be dispatched and all in-progress workflow executions became stuck. The database server itself remained healthy throughout the incident, with normal CPU and connection levels. The problem was exclusively lock and latency contention within the scheduling layer. A service restart cleared the crashed processes and temporarily restored execution. However, because the index bloat was still present, the same degradation recurred once query load resumed. A manual index rebuild fully restored performance and resolved the issue. ## Actions to Prevent Recurrence 1. **Automated index maintenance added** – Scheduled automatic index maintenance has been configured for the affected indexes. This ensures bloat cannot accumulate over time and eliminates the conditions that triggered this incident. 2. **Legacy indexes removed** – Unused legacy database indexes have been identified and removed, reducing the overall maintenance surface and simplifying future index hygiene. 3. **Monitoring coverage updated** – Our monitoring landscape is being reviewed and updated to reflect the current state of the workflow engine. This work will close gaps that allowed the degradation to go undetected before the first customer report.

Between June 16, 2026 01:00PM CEST and June 17, 2026 11:00AM CEST, the Phrase Job Sync connector suffered degraded performance. Customers using the Job Sync with the default connection type experienced failures. The engineering team identified a root cause and provided a fix.

### Introduction We would like to share more details about the events that occurred with Phrase between June 16, 2026, 01:00PM CEST and June 17, 2026, 11:00AM CEST, which led to degraded performance of Phrase Strings API, causing customers using the Job Sync with the default connection type to experience failures. We apologize for the disruption and are committed to preventing similar incidents in the future. ### Timeline **Jun 16, 2026 @ 01:00 PM CEST** – Job Sync began failing for customers using the default connection type. Connectors were unable to authenticate against the Phrase Strings API. **Jun 17, 2026 @ 10:00 AM CEST** – The authentication failure was identified by the engineering team. **Jun 17, 2026 @ 10:28 AM CEST** – Impact scope was confirmed: Customers using the default Job Sync connection were affected. Customers using personal access token-based connectors were not affected. **Jun 17, 2026 @ 10:29 AM CEST** – Root cause was identified: A recent change which added multi-platform token support inadvertently introduced errors with some existing platform tokens. **Jun 17, 2026 @ 10:40 AM CEST** – A fix was prepared and submitted for review. **Jun 17, 2026 @ 10:53 AM CEST** – The fix was deployed and JobSync functionality was fully restored. ### Root Cause A code change introduced to add support for multi-platform authentication tokens modified how incoming platform tokens are validated and processed in the Phrase Strings API. This change was not compatible with an existing token format used by the default connection type. As a result, platform tokens were rejected by the Strings API with a `401 Unauthorized` response. The connector retried establishing the connection until exhausting its retry budget, causing all affected operations to fail. Customers using personal access tokens were not affected as their tokens follow a different code path. ### Actions to Prevent Recurrence The root cause was a backwards-incompatible change to token handling that was not detected during development. The following actions are being taken: * **Token format compatibility testing in auth changes:** When introducing a new token/authentication format alongside an existing one, tests must explicitly cover the transition case—old-format tokens being processed under the new detection logic—not just each format in isolation.