Performance Disruption of Phrase Orchestrator (EU) Workflow Engine component between December 6, 2024 9:19 PM CET and December 6, 2024 11:28 PM CET

Update timeline

1. resolved Dec 09, 2024, 03:57 PM UTC

   Orchestrator (EU) Workflow Engine component experienced an outage between 9:19 PM and 11:28 PM CET. The incident has been solved.

2. postmortem Dec 13, 2024, 02:50 PM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between 9:19 PM CET and 11:28 PM CET on December 6, 2024 which led to a gradual outage of the Orchestrator Workflow Engine component and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:19 PM CET: A Phrase engineer received multiple alerts that a system component had become unavailable. 09:45 PM CET: Other Phrase engineers were notified for support. 09:53 PM CET: Phrase engineers confirmed large workloads had consumed available memory resulting in system unavailability. 10:47 PM CET: Phrase engineers began increasing memory resources in an attempt to restore system availability. 10:54 PM CET: Phrase engineers applied further memory resources. 11:17 PM CET: Phrase engineers took steps to halt a workload that was suspected as having consumed large amounts of memory. 11:28 PM CET: The system was confirmed to be operating normally. ‌ **Root Cause** The system ran under an unusually heavy load due to large workloads resulting in the system being overwhelmed due to a lack of memory resources. ### **Actions to Prevent Recurrence** A review of system resources and configuration related to the affected component is being conducted.