System latency observed in US Swimlanes Production Environment

Update timeline

1. resolved Mar 20, 2026, 04:18 PM UTC

   The CSOD Technology Team observed a performance degradation affecting US Swimlanes Production Environment (Start time 8:05 am PST to End time 8:15 am PST). During this time, clients with portals on these swimlanes may have experienced slow performance or intermittent errors while accessing the application. Our Engineering teams have implemented a fix and after a period of monitoring we are considering this issue resolved.

2. postmortem Apr 21, 2026, 06:17 PM UTC

   **Incident Summary:** Since March 11th, 2026, clients have been experiencing intermittent timeouts while accessing portals across US Swimlanes \(US SL1/2/3/5\). **Root Cause Analysis \(RCA\):** The issue was attributed to congestion in the network layer, resulting in packet drops and intermittent request failures. Increased traffic load exposed capacity and routing inefficiencies across network and application tiers. ‌ **Progressive Updates & Actions Taken:** **March 11th, 2026:** * Initial issue observed with intermittent timeouts across multiple swimlanes. * Preliminary analysis indicated packet drops due to network congestion. **March 19th, 2026:** * Tune health checks and optimize scaling behavior to improve responsiveness during sudden traffic spikes. * Introduce scheduled scaling to align capacity with peak and off-hours demand. **March 26th, 2026:** * Increased number of gateway instances to distribute traffic load more effectively. * Initial improvement observed in request handling capacity. **April 1st, 2026:** * Network device configurations were refreshed to stabilize traffic flow. * Reduced packet drops observed post configuration updates. **April 7th, 2026:** * Routing updates were implemented for specific high-volume API calls to optimize traffic paths. * Latency improvements observed for targeted request flows. **April 11th, 2026:** * Network layer capacity was enhanced by adding additional servers to better handle incoming traffic. * Improved stability and reduced timeout occurrences across application tier. **Preventive Actions/Ongoing Improvements:** * Continue horizontal scaling of gateway and application components based on traffic patterns. * Periodic review and optimization of network routing and configurations. * Ongoing infrastructure upgrades to significantly improve overall throughput and resiliency. **Resolution Summary \(Post April 11th, 2026\):** Over the weekend, capacity across key network components was further enhanced after confirming congestion-related packet drops. Additionally, required registration updates and configuration changes were implemented to optimize traffic handling and improve request reliability across the network. Following these changes, system stability improved significantly, with no widespread timeout patterns observed.