Intermittent Reporting 2.0 Issues on US PROD Across Swimlanes

Update timeline

1. resolved May 28, 2026, 04:37 PM UTC

   A severe production incident occurred in the US PROD environment, causing intermittent issues with Reporting 2.0 services across multiple swimlanes. The CSX application was impacted, experiencing degraded performance and potential data inconsistencies. Our engineering teams identified the issue and deployed a fix. We closely monitored the situation to ensure stability. After a period of monitoring with no further recurrences, the issue has been resolved.

2. postmortem Jun 23, 2026, 04:28 AM UTC

   **Incident Summary:** On May 28th, 2026, clients in the US Production environment \(SL1, SL2, SL3, and SL5\) experienced intermittent reporting errors for a brief duration. During the incident window, users encountered delays and intermittent failures while accessing reporting functionality, with a subset of reporting requests timing out. **Root Cause Analysis \(RCA\):** The issue was caused by a temporary capacity management and workload distribution inefficiency during a period of changing demand. This resulted in increased request processing times within the reporting service, leading to intermittent reporting failures and request timeouts. **Resolution:** Service capacity was stabilized by maintaining increased server capacity, ensuring sufficient resources were available to process reporting requests. Following this change, reporting functionality returned to normal operation. **Preventive Actions:** * Increased baseline service capacity to better support workload demands * Adjusted capacity management settings to provide more consistent performance during periods of fluctuating demand