Degraded Performance of Phrase Orchestrator (EU) Workflow Builder Component between August 6, 2024 1:01 PM CEST and 1:26 PM CEST

Update timeline

1. resolved Aug 07, 2024, 10:37 AM UTC

   On August 6, 2024 at 1:01 PM CEST, an outage of the Workflow Builder occurred and was resolved by our engineers at 1:26 PM CEST the same day.

2. postmortem Aug 07, 2024, 10:39 AM UTC

   # **Root Cause Analysis** August 6, 2024 ### **Introduction** We would like to share more details about the events that occurred with Phrase between 13:01 PM CEST and 13:26 PM CEST on August 6, 2024, which led to an outage of the Workflow Builder and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 6/8/2024 13:01 PM \(CEST\): A Phrase team member communicated internally that the Workflow Builder executions page had become unavailable. 6/8/2024 13:04 PM \(CEST\): Phrase engineers began mitigation steps, with the aim of reducing database CPU load. 6/8/2024 13:07 PM \(CEST\): Phrase engineers began tracking the issue internally. 6/8/2024 13:16 PM \(CEST\): The Workflow Builder executions page and related services were deemed to have recovered. 6/8/2024 13:17 PM \(CEST\): Phrase engineers communicated the recovery internally. 6/8/2024 13:26 PM \(CEST\): Preventative measures were deployed. ### **Root Cause** Queue configuration in combination with an irregularly high volume of batch action upgrades led to excessive database CPU usage, this resulted in the inaccessibility of the Workflow Builder executions page. ### **Actions to Prevent Recurrence** A dedicated queue has been added and configured.