[Multiple Regions] Agents - Degraded performance for LLM Calls

Update timeline

1. investigating Jul 08, 2026, 05:26 PM UTC

   We are investigating reports of degraded performance impacting LLM Calls for Agents in the US and EU regions. Impact: Users may experience errors from LLM calls by Agents. Our teams are working to identify the cause and will share more details as the investigation progresses.

2. monitoring Jul 08, 2026, 05:41 PM UTC

   Mitigation has been applied and performance is improving for LLM Calls in Agents across the US and EU regions. Impact: Users may still experience failed LLM calls, but this should continue to improve. We are monitoring closely to ensure stability.

3. resolved Jul 08, 2026, 06:56 PM UTC

   The issue has been resolved and Agents performance has returned to expected levels after degraded performance impacted LLM Calls in US and Europe. Impact: No ongoing user impact.

4. postmortem Jul 14, 2026, 08:56 AM UTC

   ## Customer Impact On July 8, for about five and a half hours \(roughly 12:00 to 17:40 UTC\), a subset of Agents customers in the US and EU intermittently received HTTP 500 errors on LLM completion calls. Only calls routed to one affected instance failed, so retries often succeeded. No data was lost and no other functionality was affected. ## Root Cause The failures were caused by SNAT port exhaustion on a single node in one of our shared production clusters. That node reached its fixed limit of outbound ports, which are shared by all pods running on it, so it could no longer open new outbound connections. As a result, the agents calls to the LLM Gateway failed at the network layer and returned 500 to customers. The exhaustion was driven by the combined outbound connection volume of several services running on that node. We have identified the code that causes this problem, and are implementing a fix. ## Response Detection was delayed because of an independent under-reporting issne in telemetry, so our alerts stayed below paging severity. Once identified, the affected node was drained, which moved the pods onto healthy nodes and stopped the errors immediately. Draining a shared production node is a last-resort action, since it also evicts other teams' workloads from that node, which is why it was applied late rather than early. ## Follow-Up 1. Harden the code against SNAT leaks. 2. Create a new alert that would help us identify such issues quickly on the service sde.