Performance Disruption of Phrase Orchestrator (EU) Workflow Engine component between November 5, 2024 8:18 AM CET and November 5, 2024 09:07 AM CET

Update timeline

1. resolved Nov 05, 2024, 10:43 AM UTC

   Orchestrator (EU) Workflow Engine component experienced an outage between 8:18 and 08:37 during which no workflows could be published or unpublished. Additionally, all workflow executions were delayed until 9:07. The incident has been solved and no workflow executions are expected to have been missed.

2. postmortem Nov 07, 2024, 08:13 AM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between November 5, 2024 8:18 AM CET and November 5, 2024 9:07 AM CET which led to partial outage and degraded performance of the Orchestrator Workflow Engine component and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 5/11/2024 8:18 AM CET: A Phrase engineer received an alert that a system component had become unavailable. 5/11/2024 8:30 AM CET: Phrase’s internal team was notified. 5/11/2024 8:37 AM CET: Phrase engineers increased system resources as a first mitigation step. This resulted in the system component becoming available again. 5/11/2024 9:05 AM CET: Phrase engineers noticed the system component had not fully recovered. 5/11/2024 9:06 AM CET: Phrase engineers began restarting all services belonging to the Workflow Engine component as a final mitigation step. 5/11/2024 9:07 AM CET: The system was confirmed to be operating normally. ‌ **Root Cause** The system ran out of memory, causing it to restart. This meant it couldn't process or manage any workflows during that time. ### **Actions to Prevent Recurrence** * Permanent increase of system memory on the affected component.