Memsource Outage History

Started Feb 23, 2026, 12:27 PM UTC· resolved Feb 23, 2026, 12:27 PM UTC

1. resolved Feb 25, 2026, 12:58 PM UTC

   A temporary spike in backend load caused performance disruption and brief unavailability of the Termbase service in the EU production environment. The issue was automatically mitigated, and we are continuing our analysis and implementing preventive measures to reduce the likelihood of recurrence.

Read the full incident report →

Started Feb 05, 2026, 04:56 PM UTC· resolved Feb 05, 2026, 07:47 PM UTC

1. investigating Feb 05, 2026, 04:56 PM UTC

   Engineering has identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.

2. investigating Feb 05, 2026, 05:23 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 05, 2026, 06:59 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 05, 2026, 07:20 PM UTC

   A fix has been implemented and the system is currently stable. We are continuing to monitor the situation.

5. monitoring Feb 05, 2026, 07:20 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Feb 05, 2026, 07:47 PM UTC

   This incident has been resolved.

7. postmortem Feb 19, 2026, 09:31 AM UTC

   ## Introduction We would like to share more details about the workflow disruption that occurred on February 5, 2026, between 15:01 CET and 19:58 CET, which led to workflows being held in an “Executing” state and delays in workflow processing within Phrase Orchestrator. This affected workflows running on the new engine in the EU data center. During this time, newly triggered workflows were not progressing as expected. Below we describe what happened, the root cause, and the steps we are taking to prevent similar incidents in the future. ## Timeline **Feb 5, 2026** * **16:40 –** A Phrase employee reported that workflows for an organization were stuck in “Executing” status. * **17:05 –** Phrase engineers began investigating the issue: It was observed that the component that acts as an entry point for all workflows was unstable * **17:10** To decrease the load, the engineers decreased the amout of parallel workflow tasks * **17:41** The analysis identified a degraded communication with the the database * **19:28** Following the implementation of multiple load-relief and scalability strategies, engineers determined that the issue was solely caused by a small number of extraordinarily large workflows. * **19:58** After ruling out negative side-effects, the workflows associated with the excessive load were cancelled. * **20:01** Workflows resumed normal execution. * **20:12** Previously stuck workflow executions for the affected organizations were marked as “Failed” to ensure system consistency. ## Root Cause The incident was triggered by the execution of a very large workflow with a complex dependency structure. A workflow of this design results in rapid expansion of tasks when processing large payloads. On February 5, a workflow of this nature was triggered multiple times. Due to the complex dependency structure, this resulted in the creation of more than 140,000 tasks over a short period of time. Most of these tasks executed requests against a single API, where the API rate limit was reached. This caused a large number of tasks to retry repeatedly. At the same time, the workflow engine had to evaluate the state of many dependent tasks within the workflow graph. The system executes complex database queries to determine the state of scheduled workflow jobs and their dependencies. With tens of thousands of jobs and large dependency trees, these queries became increasingly slow. The combination of … * A very high number of generated jobs * Frequent retries due to APIs and hitting rate limiting * Complex dependency evaluation for large workflows … led to long-running database queries. These slow queries exhausted available database connections and caused repeated crashes and restarts of the multiple workflow engine components. One of the affected components was the entry point for all workflows - its instability temporarily prevented other workflows from progressing. Once the tasks related to the complex workflows were cancelled, the database load immediately decreased and normal processing resumed. ## Actions to Prevent Recurrence We are taking the following steps to prevent similar incidents in the future: * **Improved Alerting:** We are improving our alerting mechanisms to ensure we are notified more quickly when workflows stop progressing. This also includes improved visibility around system exceptions. * **Enhanced Monitoring:** We are expanding our monitoring around workflow executions to more quickly identify large workloads that may clog the system. * **Workflow Architecture Redesign:** We are redesigning how workflows with complex dependency structures are handled. Complex workflow segments will be encapsulated into separate entities, reducing dependency tree complexity and improving overall processing efficiency. * **Dedicated Database Connection:** We are separating the workflow engine from the main application at the database connection level. The engine will use a dedicated connection with appropriate capacity, improving flexibility and ensuring better isolation between components. * **Improved API Rate Limit Handling:** We are enhancing how we handle APIs and how we react when rate limits get reached. * **Faster Mitigation:** We are establishing tooling to more quickly resolve workflow executions that are no longer progressing as expected. We sincerely apologize for the disruption caused by this incident. We are committed to improving the resilience and predictability of our workflow processing system and appreciate the feedback and patience of our customers.

Read the full incident report →

Started Jan 29, 2026, 04:51 PM UTC· resolved Jan 29, 2026, 05:45 PM UTC

1. identified Jan 29, 2026, 04:51 PM UTC

   We've identified performance issues in Phrase TMS CAT Web Editor and Project Management. We've identified the root cause and we are working on a fix.

2. monitoring Jan 29, 2026, 04:56 PM UTC

   A fix has been deployed and we are monitoring the results.

3. resolved Jan 29, 2026, 05:45 PM UTC

   Affected components are now stable. The incident has been resolved.

4. postmortem Feb 23, 2026, 07:13 AM UTC

   ## Introduction On January 29, 2026, between 4:56 PM and 5:46 PM CET, customers using the Phrase TMS UI in the EU region experienced repeated logouts and instability. API access was not affected. The incident was classified as SEV-1 and was resolved the same day. ## Timeline * **4:34 PM CET** – Deployment of the TMS core application in the EU region started, including a change to session storage. * **~5:00 PM CET** – Customers began reporting repeated logouts and UI issues. * **~5:30 PM CET** – The team identified inconsistent session storage configuration across application instances during the rollout. * **~5:48 PM CET** – Deployment completed across all instances. * **5:46 PM CET** – Customer impact ended. * **6:45 PM CET** – Status page incident closed after monitoring confirmed stability. ## Root Cause During a deployment, session storage was migrated to a new caching solution. Not all application instances were updated at the same time. As a result, different instances were using different session stores. Since requests from the same user can be handled by different instances, some sessions could not be found, leading to repeated logouts in the UI. API usage was not impacted. ## Actions to Prevent Recurrence * Treat session storage changes as planned maintenance and communicate them in advance. * Apply stricter deployment controls when modifying shared components. * Update internal guidelines to classify session storage configuration changes as high-risk.

Read the full incident report →

Started Jan 27, 2026, 12:54 PM UTC· resolved Jan 27, 2026, 08:11 PM UTC

1. identified Jan 27, 2026, 12:54 PM UTC

   Our engineering team has identified the root cause as an LQA-related issue and is actively working to resolve it. The project creation and configuration pages may be unavailable during this time.

2. identified Jan 27, 2026, 01:57 PM UTC

   A fix is still under active development by our engineering team.

3. identified Jan 27, 2026, 02:34 PM UTC

   A fix is still under active development by our engineering team. Project Management component is more stable now.

4. monitoring Jan 27, 2026, 05:57 PM UTC

   A fix has been implemented. Project management is stable and we are monitoring performance.

5. resolved Jan 27, 2026, 08:11 PM UTC

   The issue has been fixed. Project management is stable.

6. postmortem Feb 23, 2026, 07:09 AM UTC

   ## Introduction We would like to share more details about the events that occurred on January 27, 2026, between approximately 10:00 AM UTC and 6:30 PM CET, which led to a performance disruption of the Phrase TMS \(EU\) Project Management component. During this time, some users experienced slow or unresponsive behavior when creating or editing projects. The issue was caused by limitations in the underlying database connections used by the LQA service, which is internally used by the Project Management component. ## Timeline \(UTC\) **Jan 27, 2026 @ ~10:00 AM** First customer reports indicate slow or unresponsive project creation and editing in the EU region. **10:54 AM** Automated alert triggered due to a high number of active backend sessions in the EU production environment. **11:00–11:30 AM** Engineering investigation begins. Logs indicate that the LQA service is unable to obtain database connections, with repeated timeouts when requesting connections from the application’s connection pool. **11:40 PM** Application logs show “Too many connections” errors from the underlying database. Some service instances restart as a result. **12:00–3:00 PM** Initial mitigation steps taken: * Increase of database connection limits. * Increase of application-side connection pool limits. * Additional monitoring and logging enabled. The issue improves but intermittent errors persist. **3:30 PM** Decision made to scale the underlying production database instance to a larger size and adjust connection limits accordingly. **~5:30 PM** Database scaling completed. Error rates drop and no further “Too many connections” errors are observed. System performance stabilizes. Incident marked as resolved after continued monitoring confirmed stable behavior. ## Root Cause The disruption was caused by exhaustion of available database connections in the LQA service’s underlying production database. The LQA service uses a connection pool to communicate with the database. Under increased load, the configured connection limits on both the application side and the database side were insufficient. As more requests were processed, all available database connections were consumed. Once the limit was reached: * New requests could not obtain a database connection. * Requests timed out after waiting for a free connection. * Some service instances restarted due to repeated failures. * Project creation and editing operations became slow or temporarily unresponsive. Although the database itself was operational, the maximum number of allowed concurrent connections was too low for the actual usage patterns in production. Additionally, the size of the database instance limited how many connections could be supported safely. This combination led to a bottleneck in the LQA service, which in turn affected the Project Management component in the EU region. ## Actions to Prevent Recurrence To reduce the likelihood of similar incidents in the future, we are implementing the following measures: 1. **Database Capacity Increase** The production database instance for the affected service has been scaled to a larger size to support higher load and more concurrent connections. 2. **Improved Monitoring and Visibility** Additional database performance monitoring has been enabled to provide better insight into: * Active connections * Slow queries * Resource utilization 3. **Resilience Improvements in Project Management** We have initiated follow-up work to improve system resilience so that if the LQA service becomes slow or temporarily unavailable, it does not fully block project creation or editing operations.

Read the full incident report →

Started Jan 15, 2026, 05:38 PM UTC· resolved Jan 15, 2026, 06:50 PM UTC

1. investigating Jan 15, 2026, 05:38 PM UTC

   Phrase Portal is currently not accessible. Our engineering team is investigating the issue.

2. identified Jan 15, 2026, 06:04 PM UTC

   Engineers identified the cause of the issue and will deploy a fix.

3. identified Jan 15, 2026, 06:15 PM UTC

   A fix has been deployed and now Phrase Portal is accessible. We continue to monitor performance.

4. monitoring Jan 15, 2026, 06:15 PM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Jan 15, 2026, 06:22 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Jan 15, 2026, 06:50 PM UTC

   This incident has been resolved.

7. postmortem Jan 23, 2026, 04:07 PM UTC

   ### Introduction We would like to share details about an incident that affected the availability of the RawMT UI portal between **4:15 PM and 6:03 PM UTC** on January 15, 2026. During this time, users were unable to access the portal. The disruption was caused by a capacity limit being exceeded in our routing infrastructure. This post-mortem outlines what occurred and the steps we are taking to prevent similar issues in the future. ### Timeline **Jan 15, 2026 @ 4:15 PM UTC** – A high-severity alert was triggered as the RawMT UI portal became inaccessible to users. **Jan 15, 2026 @ 4:20–5:00 PM UTC** – Engineering teams identified that the issue was related to the routing layer reaching a fixed platform limit, preventing service traffic from being properly directed. **Jan 15, 2026 @ 5:03 PM UTC** – A mitigation was applied, re-routing affected services through alternate infrastructure. **Jan 15, 2026 @ 6:03 PM UTC** – Service was confirmed fully restored and the incident was marked as stable. ### Root Cause The issue occurred when the platform’s traffic routing infrastructure reached a fixed capacity limit on the number of active service associations it could manage. Once this limit was exceeded, configuration updates failed, preventing traffic from being correctly routed to certain services. Although application services remained operational, they became unreachable to users because there were no valid routing paths available. This resulted in a complete outage of the RawMT UI portal. ### Actions to Prevent Recurrence 1. **Improved Alerting** Alert thresholds for routing capacity have been adjusted to ensure earlier visibility and to elevate urgency before reaching critical limits. 2. **Service Distribution Across Infrastructure** Services have been actively rebalanced across multiple routing layers to reduce pressure on any single entry point. 3. **Planned Architecture Improvements** Work is underway to evaluate longer-term changes to the routing architecture to better support scale and reduce the risk of reaching fixed limits in the future.

Read the full incident report →

Started Jan 09, 2026, 04:17 PM UTC· resolved Jan 09, 2026, 04:56 PM UTC

1. investigating Jan 09, 2026, 04:17 PM UTC

   Newly published workflows may experience delayed execution. We are actively investigating the issue.

2. monitoring Jan 09, 2026, 04:50 PM UTC

   We have restored partial functionality; the team is closely monitoring the system while exploring a permanent fix.

3. resolved Jan 09, 2026, 04:56 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 29, 2025, 08:45 AM UTC· resolved Dec 29, 2025, 07:00 PM UTC

1. investigating Dec 29, 2025, 02:37 PM UTC

   The CAT Web Editor is currently experiencing long loading times. Our engineers are investigating the issue.

2. monitoring Dec 29, 2025, 06:26 PM UTC

   A fix has been implemented, and we are monitoring the situation.

3. resolved Dec 30, 2025, 08:25 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 10, 2025, 11:42 AM UTC· resolved Dec 15, 2025, 10:06 AM UTC

1. investigating Dec 10, 2025, 11:42 AM UTC

   Some users might still be experiencing issues with term bases in Phrase TMS.

2. identified Dec 10, 2025, 04:21 PM UTC

   The root cause of the issue has been identified, and work on a fix is currently underway.

3. identified Dec 11, 2025, 09:50 AM UTC

   A fix is still under active development by our engineering team.

4. identified Dec 11, 2025, 04:28 PM UTC

   Testing and preparation of the mitigation fix for release are still underway.

5. identified Dec 12, 2025, 08:38 AM UTC

   The fix is still in progress.

6. monitoring Dec 12, 2025, 02:18 PM UTC

   The fix has been deployed, and system metrics will continue to be observed.

7. resolved Dec 15, 2025, 10:06 AM UTC

   Performance has stabilized following the deployment of the fix, and the incident has been resolved.

8. postmortem Dec 19, 2025, 03:28 PM UTC

   ### Introduction We would like to share more details about a performance degradation that affected the Term Base service between December 9 and December 15, 2025. During this period, customers experienced issues where the Term Base service was intermittently unavailable or significantly delayed in processing requests. This disruption impacted users relying on Term Bases across various workflows. The engineering team has resolved the issue and is implementing further improvements to prevent similar events in the future. ### Timeline **Dec 9, 2025 @ 04:10 CET** – Term Base service began experiencing intermittent unavailability, with requests failing or timing out. On-call engineers were alerted and began investigation. **Dec 10, 2025 @ 08:30 CET to Dec 10, 2025 @ 11:40 CET** – Degraded performance was observed with the Term Base service. The team initially suspected an internal processing job in the database, which was paused. During this time, database load was found to be elevated and the database was scaled out to support the investigation. **Dec 11, 2025 @ 05:20 CET to Dec 11, 2025 @ 07:40 CET** – Partial outage occurred, during which approximately 350 Term Base API requests were not processed. The team identified that a specific pattern of search requests for large Term Bases was responsible for the disruption. A mitigation was deployed to timeout slow requests while work on a permanent solution continued. **Dec 12, 2025 @ 15:45 CET** – A fix was validated in the test environment and deployed to production. **Dec 15, 2025 @ 19:47 CET** – After continued monitoring confirmed stability, the incident was officially marked as resolved. ### Root Cause The performance degradation was caused by a combination of factors: * A very high volume of incoming requests targeting the Term Base Search API. * These requests included unusually long search texts and targeted large Term Bases. * The resulting processing demands led to resource exhaustion on backend servers, affecting availability and response times. These request patterns overwhelmed the system’s ability to respond efficiently, leading to cascading delays and failed API calls during peak periods. ### Actions to Prevent Recurrence * The logic for handling complex Term Base search requests has been optimized to reduce processing load and improve performance. * The engineering team is working on architectural improvements to isolate the impact of expensive requests and improve overall system resilience. * Monitoring and alerting thresholds for high-frequency, high-cost request patterns are being refined to allow earlier detection and mitigation.

Read the full incident report →

Started Dec 10, 2025, 07:30 AM UTC· resolved Dec 10, 2025, 07:39 AM UTC

1. resolved Dec 10, 2025, 08:31 AM UTC

   Some users might have experienced availability or performance issues with term bases in Phrase TMS.

Read the full incident report →

Started Oct 20, 2025, 01:16 PM UTC· resolved Oct 20, 2025, 03:30 PM UTC

1. monitoring Oct 20, 2025, 01:16 PM UTC

   We are aware that we have an issue across multiple Phrase services. Issue is related to the upstream partner outage. Our engineering team is actively working to ensure minimal disruption. Thank you for your patience.

2. resolved Oct 20, 2025, 03:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 10, 2025, 10:52 AM UTC· resolved Oct 10, 2025, 01:34 PM UTC

1. investigating Oct 10, 2025, 10:52 AM UTC

   Phrase Orchestrator (EU & US) has been experiencing degraded performance. The engineering team is investigating the issue. We apologize for any inconvenience caused.

2. monitoring Oct 10, 2025, 11:21 AM UTC

   Recovery is in progress across both EU and US regions, and the queues are currently being processed. Customers might still experience delayed workflow executions while the backlog is cleared. We are closely monitoring the situation to ensure full stability.

3. resolved Oct 10, 2025, 01:34 PM UTC

   This incident has been resolved.

4. postmortem Oct 14, 2025, 11:50 AM UTC

   # **Root Cause Analysis** October 10, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase Orchestrator \(EU & US\) between 10:05 AM CEST and 01:00 PM CEST on October 10, 2025 which led to delayed workflow executions for non-scheduled workflows and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 10:05 AM CEST: Our logging system showed repeated errors after introducing a change to Orchestrator’s system dependencies. 11:54 AM CEST: Through our monitoring system, Orchestrator engineers observed that workflows were not being processed by the Orchestrator workflow engine component. They began investigating and quickly identified the root cause - a recent system dependency change. 12:09 PM CEST: Our engineers prepared a rollback of the problematic change. 01:00 PM CEST: The rollback was successful. The workflow engine component began processing queued workflows again. 03:15 PM CEST: Orchestrator engineers confirmed the queue had cleared. ### **Root Cause** There was an update to a dependency which checks if the Workflow Engine component is under undue load. Due to a change in behavior of this dependency, the Workflow Builder component was unable to receive accurate load data from the Workflow Engine component. A, as a consequence, Orchestrator’s fail-safe mechanism activated to prevent data loss, causing no further workflows to execute in the workflow engine. ### **Actions to Prevent Recurrence** To prevent similar incidents in the future will improve our alerting around error logs. Additionally, we have identified further improvements for our alerting system that will notify us earlier in case of workflows not being processed. We will also speed up the process of rolling back releases in order to revert changes faster.

Read the full incident report →

Started Oct 10, 2025, 10:00 AM UTC· resolved Oct 10, 2025, 10:17 AM UTC

1. resolved Oct 13, 2025, 08:12 AM UTC

   Between 12:17 and 12:40 PM CEST on October 10, 2025, some users were unable to access Phrase TMS (EU). Service availability returned to normal by 12:40 PM CEST. No further impact has been observed since.

2. postmortem Nov 05, 2025, 08:14 AM UTC

   # **Root Cause Analysis** October 10, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase between 12:17 PM CEST and 12:26 PM CEST on October 10, 2025, which led to a performance disruption of all Phrase TMS \(EU\) components and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 12:16 PM CEST: Platform & Engineering team was alerted by the monitoring system 12:18 PM CEST: The Tech Support team was alerted to an issue affecting access to Phrase TMS \(EU\). 12:26 PM CEST: Platform engineers restarted unresponsive database nodes and increased instance size ### **Root Cause** For a few minutes, the _Custom Metadata Service_ experienced a temporary overload due to a sudden increase in load. This caused the service to become inaccessible. Because the TMS UI was incorrectly dependent on this internal service, customers using _Custom Metadata_ were unable to access the TMS during this period. Service was restored once the system load normalized. ### **Actions to Prevent Recurrence** * Updated the UI API endpoint to remove the dependency on the Custom Metadata Service. TMS will now remain accessible, with only Custom Metadata functionality limited during similar conditions. * Increased the database node instance size to better handle sudden spikes in resource utilization and prevent similar overloads.

Read the full incident report →

Started Sep 16, 2025, 11:35 AM UTC· resolved Sep 16, 2025, 12:04 PM UTC

1. identified Sep 16, 2025, 11:35 AM UTC

   We are aware of issues in the CAT web editor, where parts of the interface may appear hidden. Our engineering team found the root cause and a fix is being deployed. The resolution is expected to complete within 30 minutes.

2. resolved Sep 16, 2025, 12:04 PM UTC

   This incident has been resolved; if the CAT web editor still isn't loading, please clear your browser cache and cookies.

3. postmortem Sep 17, 2025, 06:36 PM UTC

   # **Root Cause Analysis** September 16, 2025 ‌ **Introduction** We would like to share more details about the events that occurred with Phrase between 12:45 PM CEST and 13:45 PM CEST on September 16, 2025 which led to a performance disruption of Phrase TMS \(EU and US\) CAT Web Editor Component and what Phrase engineers are doing to prevent these issues from reoccurring. ‌ **Timeline** Sep 16, 2025 @ 12:45 PM – Deployment of TMS core component was delivered to the production environment. ‌ Sep 16, 2025 @ 12:56 PM – The CAT editor team was alerted to an issue affecting the CAT web editor. Sep 16, 2025 @ 13:12 PM – Engineers identified an incompatibility issue between two components being deployed to production shortly after each other. Sep 16, 2025 @ 13:30 PM – Duration of one component's deployment prolonged the restoration of correct UI behavior. ‌ Sep 16, 2025 @ 13:45 PM – All deployments were finished, resolving the original issue for customers. ‌ **Root Cause** To improve browser rendering \(via CSS1Compat aka standard mode\), changes were made to the CAT editor page. These changes set the default editor window height to a very small value which was corrected by the styling rules in another component. Delivery of the second component, which included the style fix, was delayed. This caused users opening the CAT Editor during that timeframe to encounter an unusable UI with barely visible content. Once the second component successfully deployed, the issue was resolved. Some customers experienced temporary UI issues due to cached browser artifacts which were fixed by refreshing or clearing the browser cache. ‌ **Actions to Prevent Recurrence** Improved deployment flexibility - we're heavily reducing deployment time of related components to enable quicker recovery when issues are detected. Enhanced automated testing - we will evaluate how to integrate UI scenarios like CAT Editor loading into automated testing in canary environments prior to the final stage of production deployment. Frontend modernization - as part of an ongoing modernization effort, we are completely decoupling UI components to ensure future changes are self-contained and more resilient.

Read the full incident report →

Started Aug 26, 2025, 09:03 PM UTC· resolved Sep 02, 2025, 12:57 PM UTC

1. investigating Aug 26, 2025, 09:03 PM UTC

   Users might be experiencing delays in the execution of Orchestrator workflows hosted in the EU data center. Our engineers are investigating the situation.

2. investigating Aug 27, 2025, 03:44 PM UTC

   Users may still experience occasional delays in Orchestrator workflow executions. We are continuing to investigate this issue.

3. identified Aug 28, 2025, 03:43 PM UTC

   We have identified the issue causing intermittent delays in workflow executions and are working on optimizing the execution queue. While performance has been relatively stable, some users may still experience occasional delays in Orchestrator workflow executions. Our team continues to closely monitor the situation. If you encounter significant delays, please reach out to Phrase Technical Support.

4. monitoring Aug 29, 2025, 11:55 AM UTC

   Workflow executions are operating as expected, and no further delay spikes have been observed in the past 24 hours. We will continue to monitor the situation closely.

5. monitoring Sep 01, 2025, 01:32 PM UTC

   We have implemented scalability improvements designed to eliminate further delays in workflow executions. Performance is stable, and we will continue to monitor the situation closely.

6. resolved Sep 02, 2025, 12:57 PM UTC

   Since implementing scalability improvements, workflow executions have remained stable and performance is back to normal. These changes also make the system more robust in handling future workload spikes.

Read the full incident report →

Started Aug 13, 2025, 08:16 AM UTC· resolved Aug 14, 2025, 08:50 AM UTC

1. investigating Aug 13, 2025, 08:16 AM UTC

   On August 12, 2025 5:45 PM CEST we began experiencing delays in the execution of Orchestrator workflows hosted in the EU Data Center. Our engineers are currently investigating the root cause. We apologize for any inconvenience this may have caused.

2. monitoring Aug 13, 2025, 10:01 AM UTC

   The engineers have resolved the issue causing Orchestrator workflows to remain “stuck”. Previously untriggered workflows are now beginning to reprocess and should execute as expected. Please note that while workflows are no longer stuck, the processing queue will take some time to work through the backlog. As a result, your workflow may not run immediately and could take several hours to complete. Thank you very much for your patience.

3. resolved Aug 14, 2025, 08:50 AM UTC

   This incident has been resolved. Previously untriggered Orchestrator workflows in the queue have been processed and executed as expected.

4. postmortem Aug 18, 2025, 11:02 AM UTC

   # **Root Cause Analysis** August 13, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase between August 12, 2025 05:45 PM CEST and August 13, 2025 10:40 AM CEST, which led to a degraded performance of the workflow engine component of Phrase Orchestrator \(EU DC\) and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** **August 12, 2025 05:45 PM CEST**: The executed workflow throughput started to decrease, so workflows started more slowly than usual. The workflow scheduling for execution was unaffected. **August 12, 2025 07:53 PM CEST**: First external report of the issue received. **August 12, 2025 08:31 PM CEST**: Workflow processing significantly impacted. **August 13, 2025 10:40 AM CEST**: Issue identified and fix deployed. Pending workflows resumed execution, working off the queue. **August 13, 2025 07:38 PM CEST**: All delayed workflows completed. Normal operations restored. ### **Root Cause** A change was implemented into the Orchestrator’s message handling logic. This change introduced a mechanism where the workflow engine had to explicitly acknowledge receipt of each message before new messages were submitted for execution. This change was initially introduced to prevent the workflow engine from being overloaded, in cases of rapid workflow trigger executions. Due to a bug in this new acknowledgment logic, some acknowledgments were not properly registered when facing production traffic. As a result, the system incorrectly assumed that the engine was at full capacity, and gradually reduced the amount of messages sent, even though there was spare capacity on the engine. The system did not automatically notify the team about the issue, since the engine was not overloaded at the infrastructure level - the condition that would normally trigger an alert and page the on-call engineer. Importantly: * **No data or messages were lost**. * Once the rollback was completed, all queued workflows were processed successfully, though with significant delays. ### **Actions to prevent recurrence and improve time to resolution** 1. Improve detection and alerting for slow message processing, even when the engine is still responsive: currently, the system only alerts if the workflow engine infrastructure is actually overloaded. 2. Investigate a more resilient acknowledgment system. The acknowledgment system has been implemented to prevent the workflow engines overload in race condition scenarios; however, a new system must ensure that they are properly registered in all cases.

Read the full incident report →

Started Aug 06, 2025, 06:20 PM UTC· resolved Aug 06, 2025, 06:42 PM UTC

1. investigating Aug 06, 2025, 06:20 PM UTC

   We are investigating the issue.

2. monitoring Aug 06, 2025, 06:33 PM UTC

   The incident has been resolved, we are monitoring the system.

3. resolved Aug 06, 2025, 06:42 PM UTC

   The incident has been resolved.

4. postmortem Aug 31, 2025, 01:43 PM UTC

   # **Root Cause Analysis** August 6, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase between 19:45 PM CEST and 20:40 PM CEST on August 6, 2025 which led to a performance disruption of all our TMS services in the EU environment and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** Aug 6, 2025 @ 19:45 – The Platform and Application teams were alerted to database issues. Aug 6, 2025 @ 20:00 – Engineers identified that a majority of database replicas were unresponsive. TMS services performance disruption is confirmed. Aug 6, 2025 @ 20:00 – The first replica node was restarted. Aug 6, 2025 @ 20:05 – The restarted node appeared online, but a significant imbalance in resource usage was observed on the primary node as there were too few healthy nodes. Aug 6, 2025 @ 20:10 – The second replica node was restarted. Aug 6, 2025 @ 20:15 – Both replica nodes restarted but were unable to connect to the primary node. Aug 6, 2025 @ 20:15 – Customer complaints were reported, prompting escalation to the next level of engineering support. Aug 6, 2025 @ 20:25 – Investigation confirmed a higher than expected number of connections to the primary node. Aug 6, 2025 @ 20:35 – The maximum number of allowed connections on the primary node were increased, allowing both replicas to reconnect. Aug 6, 2025 @ 20:40 – Replica nodes completed resynchronization, restoring normal service. ‌ ### **Root Cause** A recent routine upgrade of the database replicas introduced an unexpected change in how system memory was allocated under high connection loads. While this behavior did not appear in test environments, in production it caused the replica nodes to run out of memory. As a result, they became unresponsive and failed to synchronize with the primary database which caused their removal from load-balancing while increasing the load on the primary node. ‌**Actions to Prevent Recurrence** As an immediate action we upscaled our database cluster - We’ve significantly increased the capacity so the cluster can handle higher demand without running into memory limits. Optimized memory usage - We’ve reduced how much memory the database requires for normal operations, making it more efficient and less prone to memory starvation under high load. Reviewed how memory is allocated - We have fine-tuned memory allocation for our database cluster resulting in minimal fragmentation of the address space and improving memory allocation performance as well. Enhanced monitoring - We’ve added extra monitoring metrics that alert us earlier if any database memory issues start to appear. Improve resilience of the database layer - We’re working on updating configuration and limits of our database layer to be more resilient in similar situations to lower any possible impact. Improved testing processes - We’re working with our application teams to include the database in more realistic load testing in lower \(non-production\) environments. This will help us catch potential issues before they ever reach customers.

Read the full incident report →

Started Jul 30, 2025, 12:00 PM UTC· resolved Jul 30, 2025, 12:00 PM UTC

1. resolved Aug 01, 2025, 01:36 PM UTC

   Phrase Orchestrator (EU) has been experiencing degraded performance. The engineering team has applied a fix and is currently monitoring the situation. We apologise for any inconvenience caused.

Read the full incident report →

Started Jun 27, 2025, 12:00 AM UTC· resolved Jun 27, 2025, 05:40 AM UTC

1. resolved Jul 03, 2025, 03:54 PM UTC

   The issue has been resolved.

2. postmortem Jul 14, 2025, 05:03 PM UTC

   # **Root Cause Analysis** July 2, 2025 ### **Introduction** We would like to share more details about the events that occurred between 2:00 AM June 27 and 6:40 AM CET on June 28, 2025, which led to missed execution triggers and also a delay relative to typical workflow starting times. ‌ Missed execution triggers could be observed between: June 27, 03:37 PM CET to 03:55 CET and from June 27, 11:30 PM CET to June 28, 0:45 AM CET. Below is a timeline of events and the steps our engineering team is taking to prevent similar issues in the future. ### **Timeline** June 27, 2:00 AM CET: Monitoring showed high traffic spikes, which were still handled by the system gracefully. June 27, 03:37 PM CET: The traffic spikes increased and the engineering team received alerts that a system component had become unstable. Phrase engineers identified a large consumption in available memory, resulting in system instability. As a first mitigation the Orchestrator Workflow Engine component was restarted. June 27, 03:52 PM CET The component up again and continued to process scheduled workflows. June 27, 11:30 PM CET: The Orchestrator Workflow Engine component became overloaded again. New workflows were no longer being triggered. June 28, 12:04 AM CET: Phrase engineers took steps to halt a workload that was suspected as having consumed large amounts of memory. ‌ June 28, 12:45 AM CET: Phrase engineers applied further memory resources and the system continued again to trigger and process scheduled workflows. June 28, 6:40 AM CET: The backlog of workflows had been completely processed. ‌ **Root Cause** ‌ The incident was caused by sustained memory pressure on the Orchestrator Workflow Engine component, triggered by a spike in incoming events caused by a single workflow. This eventually prevented the system from processing new workflows. ### **Actions to Prevent Recurrence** A thorough review of the affected system’s resource configuration is being conducted. As part of this effort, the following improvements are implemented: * **Limit radius of impact of single workflows:** We will implement mechanisms to isolate resources to reduce the likelihood of cross workflow impact. * **Proactive detection:** We will introduce additional alerts to improve early detection, to allow us to take action before customers are impacted. * **Improved Scalability:** We are adding additional scaling mechanisms to key components of the Workflow Engine to handle sudden increases in workload. We are adjusting memory limits and requests for the impacted component to ensure greater stability under load.

Read the full incident report →

Started May 20, 2025, 08:06 AM UTC· resolved May 20, 2025, 08:51 AM UTC

1. investigating May 20, 2025, 08:06 AM UTC

   Our engineers are currently investigating the root cause. We apologize for any inconvenience caused.

2. monitoring May 20, 2025, 08:18 AM UTC

   The fix has been implemented and we are monitoring the results.

3. resolved May 20, 2025, 08:51 AM UTC

   The issue has been resolved.

4. postmortem May 26, 2025, 08:40 AM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between 09:52 AM CEST and 10:14 AM CEST on May 20, 2025, which led to a performance disruption of Strings EU and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:52 AM CEST: The Strings platform became unavailable due to a change in network configuration. Investigation began immediately. 10:14 AM CEST: A temporary fix was applied, restoring full platform availability. 03:43 PM CEST: A full fix addressing the underlying issue was deployed. ‌ **Root Cause** The issue was caused by a network configuration change introduced as part of a routine Kubernetes infrastructure upgrade. After the upgrade, some application components failed health checks, which led to automatic restarts and unavailability - even though the services themselves were actually running. This happened only in cases where enhanced network security rules were applied to individual parts of the system. The way the Kubernetes and underlying network handled traffic in this setup led to health checks not working as expected. The problem was resolved by adjusting the network settings on the underlying infrastructure network stack to support these health checks properly in this configuration. ‌ **Actions to Prevent Recurrence** We have updated our infrastructure configuration to always include this network setting going forward. This ensures that future upgrades will not trigger the same issue. We will also plan and roll out future Kubernetes upgrades in a more granular manner, targeting smaller segments of infrastructure first and performing changes outside of peak traffic hours. This will help us reduce the risk of platform-wide impact and detect potential issues earlier in the rollout process.

Read the full incident report →

Started Apr 23, 2025, 04:47 AM UTC· resolved Apr 23, 2025, 07:03 AM UTC

1. investigating Apr 23, 2025, 04:47 AM UTC

   Users based in Asia reported issues with CAT web editor. Our engineering team is investigating the issue.

2. investigating Apr 23, 2025, 05:31 AM UTC

   We are continuing to investigate the issue.

3. identified Apr 23, 2025, 05:56 AM UTC

   The team found an issue in the index configuration of the database, and is actively trying to fix it as soon as possible.

4. monitoring Apr 23, 2025, 06:43 AM UTC

   The team deployed a fix that reduced the database load, and all clusters are running smoothly since then. We are continuing to monitor the situation.

5. resolved Apr 23, 2025, 07:03 AM UTC

   The incident has been resolved.

Read the full incident report →

Started Apr 15, 2025, 04:30 PM UTC· resolved Apr 15, 2025, 04:30 PM UTC

1. resolved Apr 15, 2025, 04:59 PM UTC

   The incident has been resolved.

Read the full incident report →

Started Apr 08, 2025, 11:00 AM UTC· resolved Apr 08, 2025, 11:11 PM UTC

1. resolved Apr 11, 2025, 03:58 PM UTC

   A change in the new CAT editor version impacted the persistence of segment adjustments for users, who had the editor open during the deployment and didn't refresh the page.

2. postmortem Apr 11, 2025, 03:58 PM UTC

   #### **Introduction** We would like to share more details about the event that occurred with Phrase at 1:15 PM CEST on April 8th, 2025, and its effects, which gradually diminished until 0:11 AM CEST on April 9th, 2025. During this period, the functionality of the CAT editor was impacted, causing issues with the persistence of some users' segment adjustments. The issue affected only users who were actively using the editor at the time of deployment and didn't follow the notification to refresh their browser. Users who opened the editor after the deployment were not impacted in any case. In this report, we will outline the timeline, the root causes, and detail the measures we are taking to prevent this and similar issues from reoccurring. #### **Timeline** * Apr 8, 2025 @ 13:22 CET - Go-live of a new CAT editor version. The persistence part of this version was unintentionally incompatible to the old client, which active sessions had in use. * Apr 8, 2025 @ 14:00 CET - Due to switching jobs or reloading the page, the affected users went down by 50%. * Apr 8, 2025 @ 14:29 CET - The Phrase Support team received the first report indicating a potential issue, however incoming reports coming in very low numbers. * Apr 8, 2025 @ 15:47 CET - As additional reports came in and resulting higher priority escalation to the development team and the team started to investigate the root cause. * Apr 8, 2025 @ 16:05 CET - The root cause was identify and the team assessed current impact on customers and mitigating actions; the support team started to advise users to reload the page. * Apr 8, 2025 @ 17:00 CET - 90% of affected users had resolved the issue. * Apr 8, 2025 @ 19:15 CET - Further hot-fixes were put on hold, due to the low remaining volume \(4%\) of affected sessions, in order to avoid further disruption. * Apr 9, 2025 @ 00:11 CET - The last instance of a failed segment save was observed. #### **Root Cause** Typically the editor team is running a ‘non breaking change' deployment approach, avoiding any kind interruption to users currently working, and thus enabling zero downtime deployments. However, a new version introduced a semantic breaking change in one of the APIs, which caused segment modifications - such as translating, editing pre-translated content, confirming, locking, and similar actions - to not be properly persistent in conjunction with the old client version. This breaking change was also not caught by the automated testing suite, as the syntactic API contract was met. As the failure occurred only under specific semantic conditions, it didn't trigger general error messages: changes appeared successful in the UI but were not actually persisted. Because of this there was no visibility to the development team, as no runtime errors, logs, or monitoring alerts were triggered in the staging environment, and also no issues were reported by users during the 30-hour canary phase. Users who opened the CAT editor after the deployment - or refreshed their session in response to the new version notification - experienced no disruption. Overall, according to our investigations a very minor percentage of all phrase customers and users have been affected. ### **Actions to Prevent Recurrence** * The CAT editor team will implement additional monitoring and alerting to identify discrepancies between inbound and outbound data modifications. * A additional automated test step will be introduced to strengthen the quality and detect potential compatibility issues, especially those who don’t break the API contract during the development process.

Read the full incident report →

Started Apr 02, 2025, 11:00 PM UTC· resolved Apr 03, 2025, 11:30 PM UTC

1. resolved Apr 03, 2025, 03:39 PM UTC

   Some users without a login session were not able to access Phrase TMS. Other components and users with a valid TMS session were not affected.

2. postmortem Apr 10, 2025, 11:48 AM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between 1:11 PM CEST and 2:40 PM CEST on April 3rd, 2025 which led to a partial outage of the TMS component \(not yet logged-in users were not able to switch to TMS from other platform applications\) and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 08:39 AM CEST: A new version of IDM \(Identity Management\) is released with an upgraded version of Spring boot 3.4.4 which includes Spring security library updates. This update contains a change in the BCrypt library implementation that limits password checks to 72 characters \(tracked as an issue under [https://github.com/spring-projects/spring-security/issues/16802](https://github.com/spring-projects/spring-security/issues/16802)\) even for existing passwords. 09:18 AM CEST: The new version is deployed in all instances and post-release checks are performed without any failures. 01:11 PM CEST: The inter-app token cache expires \(TMS → IDM\) and TMS asks for a new access token with a secret longer than 72 characters. This call ends up with _IllegalArgumentException_ for all requests and the pager duty alert is triggered. 01:30 PM CEST: Responsible teams track the issue down to the BCrypt library update and begins rotating passwords immediately in all places with more than 72 character passwords \(including other Phrase platform applications to prevent any further issues\). 02:00 PM CEST: First cohort of TMS servers is being deployed with new credentials. 02:40 PM CEST: All TMS servers are running and all \(not yet logged in\) users can again log into TMS. ### **Impact** Roughly **15% of users at a time window of 1:30 hours** may have experienced login issues, specifically when they were **logged out or had expired sessions** that required token refreshes. ### **Root Cause** An upstream library change introduced a previously undocumented limit in password verification logic, leading to authentication errors for secrets longer than 72 characters. We deployed a new version of our Identity Management \(IDM\) service that upgraded to Spring Boot 3.4.4. This was a **patch-level upgrade**, and in line with semantic versioning, was not expected to introduce breaking changes. However, the update included a change to the BCrypt library that limits password verification to the first 72 characters, even for existing secrets \([reference](https://github.com/spring-projects/spring-security/issues/16802)\). ### **Actions to Prevent Recurrence** * Adopt monthly library update cadence for better isolation. * Decouple library changes from major releases for quicker rollback. * Migrate from BCrypt to Argon2 for future-proofing password hashing. We follow semantic versioning principles and take stability seriously. This incident has reinforced our commitment to more granular and isolated testing of third-party library changes, even within patch updates.

Read the full incident report →

Started Mar 27, 2025, 08:18 AM UTC· resolved Mar 27, 2025, 08:36 AM UTC

1. investigating Mar 27, 2025, 08:18 AM UTC

   We are currently experiencing a performance disruption of all Phrase TMS (EU) components. Our engineering team is investigating the issue.

2. identified Mar 27, 2025, 08:26 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 27, 2025, 08:27 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 27, 2025, 08:36 AM UTC

   This incident has been resolved.

5. postmortem Apr 07, 2025, 12:47 PM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between 9:01 AM CET and 9:17 AM CET on March 27, 2025 which led to an outage of all Phrase TMS \(EU\) components and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:01 AM CET: The number of processed requests began to slowly decline. 09:07 AM CET: Monitoring systems alerted our on-call teams about the instability of the environment. 09:12 AM CET: Support tickets started to appear. Our engineers begin investigating the situation. 09:17 AM CET: A faulty component was identified and restarted, which stabilized the environment. ‌ **Root Cause** One of the caching components nodes became overloaded with incoming traffic. This caused dependent services to experience significant delays while waiting for responses from the cache middleware. ‌ **Actions to Prevent Recurrence** We are upgrading the caching system to a newer version and increasing the number of cache nodes to better spread the traffic and prevent network congestion.

Read the full incident report →

Started Mar 19, 2025, 11:49 AM UTC· resolved Mar 19, 2025, 03:16 PM UTC

1. investigating Mar 19, 2025, 11:49 AM UTC

   We are currently experiencing a performance disruption of all Phrase TMS (EU) components. Our engineering team is investigating the issue.

2. investigating Mar 19, 2025, 12:10 PM UTC

   We are continuing to investigate this issue.

3. investigating Mar 19, 2025, 12:36 PM UTC

   We continue to investigate and work on the issue.

4. investigating Mar 19, 2025, 01:14 PM UTC

   Our engineering team continues to investigate the issue and work on a resolution.

5. investigating Mar 19, 2025, 01:55 PM UTC

   Our team is still investigating the issue and working on a resolution.

6. identified Mar 19, 2025, 02:07 PM UTC

   A possible root cause has been found and our team is currently in the process of mitigating it.

7. monitoring Mar 19, 2025, 02:34 PM UTC

   Problems found have been isolated and the issues appear to be mitigated. Situation should be stable from now on - TMS is accessible again. The search service will currently stay disabled.

8. resolved Mar 19, 2025, 03:16 PM UTC

   The incident has now been resolved. All TMS components are functional.

9. postmortem Apr 14, 2025, 06:35 PM UTC

   # **Root Cause Analysis** March 19th, 2025 ### **Introduction** We would like to share more details about the events that occurred with Phrase between 12:21 PM CEST and 3:40 PM CEST on March 19th, 2025 which led to an outage of all Phrase TMS \(EU\) components and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** **10:00 AM CET:** A new search feature is enabled for Beta testing. **12:48 PM CET:** A user triggers a search using a specific filter combination that leads to unexpected system behavior. **1:11 PM CET:** The issue is identified and escalated internally. A broader test unintentionally increases system load. **1:12 PM CET:** Message traffic increases sharply, causing high network and CPU usage. **1:20 PM – 3:40 PM CET**: Engineering efforts are focused on identifying and mitigating performance issues across services. **3:00 PM CET:** Network congestion is identified between Phrase TMS and a key database cache. **3:15 PM CET:** Cache systems are restarted, which alleviates the performance bottleneck. **4:00 PM CET:** The new search feature is disabled. System traffic and performance return to normal. ### **Customer Impact** Between **12:21 PM and 3:40 PM CEST**, all **Phrase TMS \(EU\)** users experienced a degraded experience: * **Service Availability**: Project and job loading, as well as general performance, were significantly slowed or failed intermittently. * **Search Functionality**: Users engaging with the Beta search feature experienced delays or failures. * **Processing Delays**: Background tasks and automation workflows relying on real-time data were delayed. Scope of Impact * The disruption affected **all active users** of Phrase TMS \(EU\). * **Successful request rates** dropped to approximately **15%** during the peak of the incident. * Performance began recovering around **3:45 PM CEST**, and the Beta search feature was kept offline until a validated fix could be applied. ### **Root Cause** The incident was primarily caused by **network congestion** between the Phrase TMS application and its underlying **database cache layer**, which slowed data retrieval and affected overall system performance. This was triggered by: * A **bug** in the new search feature that created a high volume of retry traffic * **Unexpected message growth** which led to increased system load * **Spillover traffic** that strained shared infrastructure resources * **Delayed detection** of cache congestion, which postponed full recovery ### **Actions to Prevent Recurrence** #### Safe Failure Handling We’ve updated our systems to better handle edge cases in message processing, ensuring that a similar loop cannot cause systemic degradation. #### Cache Layer Improvements * Upgraded infrastructure for better network handling and tuning * Scaled out the cache to handle higher loads more efficiently * Improved automated recovery mechanisms #### Cache Traffic Monitoring New observability tools have been introduced to detect anomalies in application-to-cache communication, helping to surface latency spikes or throughput drops more proactively. #### Alerting Improvements We’ve enhanced our alerting strategy to detect queuing behavior and system delays sooner. #### Cross-System Correlation Awareness We’ve updated our incident response playbooks to ensure a more holistic, system-wide approach to troubleshooting and escalation. ### **Conclusion** Finally, we want to apologize. We know how critical our services are to your business. Phrase as a whole will do everything to learn from this incident and use it to drive improvements across our services. As with any significant operational issue, Phrase engineers will be working tirelessly over the next coming days and weeks on improving their understanding of the incident and to determine how to make changes that improve our services and processes.

Read the full incident report →