Latency/intermittent timeouts observed on CDG SL1 Production Environment

Update timeline

1. resolved Apr 27, 2026, 01:11 PM UTC

   CSOD Technology Team observed a service disruption on the CDG SL1 beginning at 4:21 AM PT, with service restored at 4:31 AM PT. During this time, some clients may have experienced intermittent timeouts. The issue has been resolved and the environment is stable.

2. postmortem May 21, 2026, 04:08 AM UTC

   **Incident Summary:** On April 27th, 2026, clients in the FR EXT SL1 environment experienced intermittent service degradation for a brief duration. The issue resulted in intermittent request failures and impacted application accessibility. **Impact:** Users experienced intermittent request timeouts and failed requests while accessing services in the impacted environment. Some requests returned timeout-related responses during the degradation period. **Root Cause Analysis \(RCA\):** The issue was related to transient instability within an infrastructure component handling request routing and traffic distribution. During the incident window, intermittent backend connectivity and delayed health state transitions contributed to request handling inconsistencies, resulting in timeout responses for a subset of requests. **Resolution:** As an immediate mitigation, service capacity was increased within the impacted routing layer to support traffic handling and stabilize request processing. Following recovery actions and stabilization of the affected infrastructure component, normal traffic flow and service performance were restored. **Preventive Measures:** The following measures have been implemented to reduce the likelihood of recurrence: * Service capacity and traffic handling configurations have been reviewed to improve stability during transient infrastructure events.