Degraded Performance of Phrase Orchestrator (EU) Workflow Engine Component between July 24, 2024 4:06 PM CEST and July 24, 2024 4:56 PM CEST

Update timeline

1. resolved Jul 29, 2024, 09:49 AM UTC

   We are currently investigating this issue.

2. postmortem Jul 29, 2024, 09:51 AM UTC

   # **Root Cause Analysis** July 24, 2024 ### **Introduction** We would like to share more details about the events that occurred with Phrase between July 24, 2024 4:06 PM CEST and July 24, 2024 4:56 PM CEST which led to a partial outage and degraded performance of the Workflow Engine component and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 24/7/2024 4:06 PM CEST: A Phrase engineer noticed a system component had become unavailable. 24/7/2024 4:07 PM CEST: Additional Phrase engineers have been notified. 24/7/2024 4:08 PM CEST: Phrase engineers gathered to respond to the issue. 24/7/2024 4:21 PM CEST: Phrase engineers increased system resources in an effective final mitigation step. 24/7/2024 4:26 PM CEST: The system is confirmed to be operating normally. 24/7/2024 4:56 PM CEST: Incident details were communicated internally. ‌ **Root Cause** The system ran out of resources. This is likely due to a combination of factors, including a large volume of stale execution records and system load at the time. ### **Actions to Prevent Recurrence** * Periodic disposal of stale execution records. * System change to optimize calls to the affected component.