Degraded Performance of Identity management - IDM (EU), potentially affecting Phrase TMS (EU) between April 3, 2025 1:11 PM CEST and April 3, 2025 2:40 PM CEST

Update timeline

1. resolved Apr 03, 2025, 03:39 PM UTC

   Some users without a login session were not able to access Phrase TMS. Other components and users with a valid TMS session were not affected.

2. postmortem Apr 10, 2025, 11:48 AM UTC

   ### **Introduction** We would like to share more details about the events that occurred with Phrase between 1:11 PM CEST and 2:40 PM CEST on April 3rd, 2025 which led to a partial outage of the TMS component \(not yet logged-in users were not able to switch to TMS from other platform applications\) and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 08:39 AM CEST: A new version of IDM \(Identity Management\) is released with an upgraded version of Spring boot 3.4.4 which includes Spring security library updates. This update contains a change in the BCrypt library implementation that limits password checks to 72 characters \(tracked as an issue under [https://github.com/spring-projects/spring-security/issues/16802](https://github.com/spring-projects/spring-security/issues/16802)\) even for existing passwords. 09:18 AM CEST: The new version is deployed in all instances and post-release checks are performed without any failures. 01:11 PM CEST: The inter-app token cache expires \(TMS → IDM\) and TMS asks for a new access token with a secret longer than 72 characters. This call ends up with _IllegalArgumentException_ for all requests and the pager duty alert is triggered. 01:30 PM CEST: Responsible teams track the issue down to the BCrypt library update and begins rotating passwords immediately in all places with more than 72 character passwords \(including other Phrase platform applications to prevent any further issues\). 02:00 PM CEST: First cohort of TMS servers is being deployed with new credentials. 02:40 PM CEST: All TMS servers are running and all \(not yet logged in\) users can again log into TMS. ### **Impact** Roughly **15% of users at a time window of 1:30 hours** may have experienced login issues, specifically when they were **logged out or had expired sessions** that required token refreshes. ### **Root Cause** An upstream library change introduced a previously undocumented limit in password verification logic, leading to authentication errors for secrets longer than 72 characters. We deployed a new version of our Identity Management \(IDM\) service that upgraded to Spring Boot 3.4.4. This was a **patch-level upgrade**, and in line with semantic versioning, was not expected to introduce breaking changes. However, the update included a change to the BCrypt library that limits password verification to the first 72 characters, even for existing secrets \([reference](https://github.com/spring-projects/spring-security/issues/16802)\). ### **Actions to Prevent Recurrence** * Adopt monthly library update cadence for better isolation. * Decouple library changes from major releases for quicker rollback. * Migrate from BCrypt to Argon2 for future-proofing password hashing. We follow semantic versioning principles and take stability seriously. This incident has reinforced our commitment to more granular and isolated testing of third-party library changes, even within patch updates.