Provation Apex is currently experiencing technical difficulties

Update timeline

1. investigating Feb 24, 2026, 06:35 PM UTC

   We are actively investigating an incident with Apex printing and previewing a note.

2. resolved Feb 24, 2026, 07:59 PM UTC

   This incident has been resolved.

3. postmortem Mar 04, 2026, 07:11 PM UTC

   Issue Resolution Report ‌ ‌ **Description:** On February 24, 2026, between 8:51AM and 9:22 AM CST, some customers experienced intermittent difficulties uploading video documentation and accessing other platform features. **Impact Summary:** On February 24, 2026, between 14:51-15:22 UTC \(8:51am-9:22am CST\), the Provation production environment experienced a 31-minute **degradation** affecting customer-facing operations. During the peak degradation window \(14:51-15:20 UTC\). The incident manifested as intermittent failures across multiple services dropping customer-facing success rates to 81% during the worst minutes. Auto-scaling triggered at 14:00 UTC \(8:00am CST\) provided initial capacity expansion. Operations executed additional mitigation measures and the system achieved gradual improvement through 15:21 UTC \(9:21am CST\) and reached full recovery at 15:22 UTC \(9:22am CST\) when customer-facing success rates returned to 98%\+. No data loss occurred, but clinical documentation workflows experienced intermittent failures during the incident window. Primary failure modes included server errors on shared APIs. **Response** The Development team coordinated mitigation activities to stabilize platform performance. This effort included isolating the affected service paths, adjusting system resources to support essential workflows, and continuously monitoring platform behavior to validate the most effective mitigation steps. Responsiveness, error patterns, and system health indicators were tracked to guide decision making during recovery. As mitigation actions took effect, performance improved which lead to restoration by 15:15 UTC. **Follow-up Tasks** ‌ * Split VideoAPI onto independent ASP-DevOps Team ‌ * Evaluate VNet Integration \+ Service Endpoints for Video API/Document API-DevOps Team ‌ * Evaluate private endpoints for Storage-DevOps Team ‌ * Evaluate connection pooling refactor-Development Team ‌ * Evaluate port monitoring and alerting thresholds-DevOps Team ‌ * Evaluate circuit breaker implementation for video endpoints-Development Team ‌ * Evaluate rate limiting for video uploads and vitals polling-Development Team ‌ * Evaluate gradual re-enable plan for disabled tenants-Development Team