Phrase incident
Degraded Performance of Phrase TMS (EU) Term Base Component between December 9, 2025 04:10 AM CET and December 15, 2025 07:47 PM CET
Phrase experienced a minor incident on December 10, 2025 affecting Term base, lasting 4d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 10, 2025, 11:42 AM UTC
Some users might still be experiencing issues with term bases in Phrase TMS.
- identified Dec 10, 2025, 04:21 PM UTC
The root cause of the issue has been identified, and work on a fix is currently underway.
- identified Dec 11, 2025, 09:50 AM UTC
A fix is still under active development by our engineering team.
- identified Dec 11, 2025, 04:28 PM UTC
Testing and preparation of the mitigation fix for release are still underway.
- identified Dec 12, 2025, 08:38 AM UTC
The fix is still in progress.
- monitoring Dec 12, 2025, 02:18 PM UTC
The fix has been deployed, and system metrics will continue to be observed.
- resolved Dec 15, 2025, 10:06 AM UTC
Performance has stabilized following the deployment of the fix, and the incident has been resolved.
- postmortem Dec 19, 2025, 03:28 PM UTC
### Introduction We would like to share more details about a performance degradation that affected the Term Base service between December 9 and December 15, 2025. During this period, customers experienced issues where the Term Base service was intermittently unavailable or significantly delayed in processing requests. This disruption impacted users relying on Term Bases across various workflows. The engineering team has resolved the issue and is implementing further improvements to prevent similar events in the future. ### Timeline **Dec 9, 2025 @ 04:10 CET** – Term Base service began experiencing intermittent unavailability, with requests failing or timing out. On-call engineers were alerted and began investigation. **Dec 10, 2025 @ 08:30 CET to Dec 10, 2025 @ 11:40 CET** – Degraded performance was observed with the Term Base service. The team initially suspected an internal processing job in the database, which was paused. During this time, database load was found to be elevated and the database was scaled out to support the investigation. **Dec 11, 2025 @ 05:20 CET to Dec 11, 2025 @ 07:40 CET** – Partial outage occurred, during which approximately 350 Term Base API requests were not processed. The team identified that a specific pattern of search requests for large Term Bases was responsible for the disruption. A mitigation was deployed to timeout slow requests while work on a permanent solution continued. **Dec 12, 2025 @ 15:45 CET** – A fix was validated in the test environment and deployed to production. **Dec 15, 2025 @ 19:47 CET** – After continued monitoring confirmed stability, the incident was officially marked as resolved. ### Root Cause The performance degradation was caused by a combination of factors: * A very high volume of incoming requests targeting the Term Base Search API. * These requests included unusually long search texts and targeted large Term Bases. * The resulting processing demands led to resource exhaustion on backend servers, affecting availability and response times. These request patterns overwhelmed the system’s ability to respond efficiently, leading to cascading delays and failed API calls during peak periods. ### Actions to Prevent Recurrence * The logic for handling complex Term Base search requests has been optimized to reduce processing load and improve performance. * The engineering team is working on architectural improvements to isolate the impact of expensive requests and improve overall system resilience. * Monitoring and alerting thresholds for high-frequency, high-cost request patterns are being refined to allow earlier detection and mitigation.