Intermittent failure on all app networking calls (including product apis, egress, storage and resolvers)

Update timeline

1. investigating Sep 15, 2025, 02:06 AM UTC

   We have identified a networking issue that is impacting fetch requests originating from forge apps. All forge-enabled products are impacted. Impact start time: Thursday, September 4th Symptoms include: - product fetch call timeout - egress fetch call timeout - storage calls timeout - resolver timeout The following errors may be present in your app logs - UND ERR CONNECT TIMEOUT - ERR SOCKET TIMEOUT We are currently investigating the root cause - seeing evidence of unstable network connections between distributed infrastructure - Packet loss - TCP resends - Eventual timeout on fetch operations We will provide another update at Sep 15 7:00 AM UTC

2. investigating Sep 15, 2025, 10:10 AM UTC

   We are continuing to investigate and mitigate the issue. Upto 0.1% of all requests originating from Forge apps could result in Connection Timeout errors We are continuing to diagnose the root cause and are attempting mitigation in affected regions. We will continue to monitor requests and report on any improvements. Next Steps: We will apply the mitigation in other regions and continue to monitor. Next update will be within 4 hours.

3. monitoring Sep 16, 2025, 12:07 AM UTC

   We identified a networking issue that increased timeouts for fetch requests originating from Forge apps. A small portion (< 0.1%) of requests resulted in Connection Timeout errors. Other symptoms included timeouts in product API requests, storage calls and resolver executions. Timeline: 2025-09-03 10:00 UTC: impact start 2025-09-15 16:56 UTC: impact mitigated We will continue to monitor our mitigation and investigate the root cause.

4. resolved Sep 19, 2025, 02:02 AM UTC

   No customer impact has been observed seen since mitigation began. We are now closing this incident.