RevolutionEHR experienced a critical incident on February 25, 2025 affecting RevolutionEHR United States, lasting 2h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 25, 2025, 10:24 PM UTC
We are aware of slowness and errors and are investigating and will provide an update
- investigating Feb 25, 2025, 10:30 PM UTC
Access to RevolutionEHR is currently unavailable, our engineers are working to restore access as quickly as possible
- identified Feb 25, 2025, 10:57 PM UTC
Our engineers have identified a potential cause and are working to resolve
- identified Feb 25, 2025, 11:27 PM UTC
As our engineering team works to restore access you may experience intermittent periods where you are able to log in. However, these may be intermittent. We will provide an update when the system is stable
- monitoring Feb 26, 2025, 12:05 AM UTC
A fix has been applied and we are monitoring the resolution. We will provide a post-mortem at some point tomorrow to provide more specific information regarding what happened and what is being done to ensure it has been fully addressed
- resolved Feb 26, 2025, 12:31 AM UTC
This incident has been resolved.
- postmortem Feb 26, 2025, 11:10 PM UTC
**Root Cause:** * On Tuesday afternoon \(2/25/2025\), a SEV-1 incident in RevolutionEHR caused a 1 hour, 41-minute outage. * A data conversion tool used to migrate existing RevolutionEHR customers to the new database exhausted allocated CPU and memory, causing the tool to stop running. * In response, we increased resource allocations to allow the tool to continue running and complete the data conversion process. * However, this led to a long-running transaction that caused database locks, which compounded over time and ultimately blocked access to the RevolutionEHR application for customers. **Remediation Actions:** * The development team is actively improving transaction handling within the offending data conversion tool to better manage large transactions._._ * The Data Conversion team will cease use of this new tool until improvements have been deployed and tested.