RevolutionEHR experienced a critical incident on March 21, 2025 affecting RevolutionEHR United States, lasting 3h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 21, 2025, 01:10 PM UTC
Our engineering team is actively investigating reports of lag when using RevolutionEHR. We will post updates as soon as they are available
- investigating Mar 21, 2025, 01:25 PM UTC
We are now experiencing a complete outage. All engineers are working to restore as quickly as possible
- investigating Mar 21, 2025, 01:57 PM UTC
We continue to investigate the cause of this issue
- identified Mar 21, 2025, 02:07 PM UTC
Our engineers are working to roll back the changes related to system maintenance that was performed overnight to return the system to the previous state and resolve the current issues. This will take approximately 30 minutes. Updates will be posted as available
- monitoring Mar 21, 2025, 02:43 PM UTC
System has been restored and we are closely monitoring the implemented solution
- resolved Mar 21, 2025, 04:26 PM UTC
This incident has been resolved.
- postmortem Mar 25, 2025, 02:03 PM UTC
**Root Cause Analysis:** Recent database optimization changes intended to improve performance inadvertently introduced adverse effects, resulting in degraded system stability and availability. Upon identifying this issue, we promptly reverted these changes, restoring the system to a stable state. **Steps to be taken:** * **Incremental Rollout of Configuration Changes:** Adopt a phased or incremental rollout strategy for significant configuration changes to limit potential impact and allow for early identification and mitigation of issues. * **Enhanced Proactive Monitoring and Alerting:** Improve our database monitoring capabilities and adjust alerting thresholds to proactively detect and respond quickly to performance degradation or instability.