Exoscale incident
Concrete AI Dedicated Inference - Requests ending with 500
Resolved after 13 hours 34 minutes, affecting CH-GVA-2 and CH-DK-2 and 1 more, on October 1, 2026.
Post-mortem Summary On October 1, 2026, during a scheduled maintenance window to migrate to a new AI router, a partial outage occurred affecting Dedicated Inference services. Customers initially received HTTP 404 errors, followed by HTTP 502 errors. The root cause was identified as missing TLS configurations in some customer deployments. Impact Requests to impacted Dedicated Inference deployments resulted in errors for approximately 1.5 hours. This impacted all zones where dedicated inference is available. Timeline 17:00 CET: Migration to the new vLLM router started. 17:15 CET: DNS change that resulted in routing issues, causing requests to end up with 404 errors. 17:30 CET: The routing issue has been fixed, and HTTP 502 errors were being returned 18:00 CET: The incident was escalated as a partial outage 18:07 CET: Mitigations are being gradually applied to fix the TLS configurations of the deployments 18:21 CET: All deployments across all zones have been successfully patched. 18:30 CET: Service back to nominal What Happened The new AI Router proxies to Dedicated inference deployments through mTLS. Some of the mTLS resources were missing for some deployments. Because of this misconfiguration, impacted deployments returned HTTP 502 errors for all inference requests. What We Learned We didn’t have a proper way to validate that all deployments had their mTLS configuration set up before the maintenance, and we incorrectly assumed this was true for all of them. What We’ve Changed We took an inventory and created the missing resources. Moving Forward Future infrastructural rollouts will be performed more gradually, progressing deployment by deployment, staged per zone. Also, we are adding more processes to ensure the invariants that should hold are true before performing this kind of maintenance.
Affected components
Update timeline
- investigating Oct 01, 2026, 04:01 PM UTC
Customers with dedicated deployments might encounter HTTP 500 errors when making inference requests.
- investigating Oct 01, 2026, 04:16 PM UTC
We discovered the issue and have already fixed some deployments. We are continuing to apply the mitigation.
- monitoring Oct 01, 2026, 04:21 PM UTC
All the deployments have been fixed, we are monitoring the situation.
- monitoring Oct 01, 2026, 04:58 PM UTC
The situation has been resolved; we will monitor it until tomorrow morning.
- resolved Oct 02, 2026, 05:34 AM UTC
We haven’t seen any more HTTP 500 errors since our last communication. The situation is nominal; we are sorry for the inconvenience. A public post-mortem will follow.
- investigating Oct 02, 2026, 11:04 AM UTC
Post-mortem Summary On October 1, 2026, during a scheduled maintenance window to migrate to a new AI router, a partial outage occurred affecting Dedicated Inference services. Customers initially received HTTP 404 errors, followed by HTTP 502 errors. The root cause was identified as missing TLS configurations in some customer deployments. Impact Requests to impacted Dedicated Inference deployments resulted in errors for approximately 1.5 hours. This impacted all zones where dedicated inference is available. Timeline 17:00 CET: Migration to the new vLLM router started. 17:15 CET: DNS change that resulted in routing issues, causing requests to end up with 404 errors. 17:30 CET: The routing issue has been fixed, and HTTP 502 errors were being returned 18:00 CET: The incident was escalated as a partial outage 18:07 CET: Mitigations are being gradually applied to fix the TLS configurations of the deployments 18:21 CET: All deployments across all zones have been successfully patched. 18:30 CET: Service back to nominal What Happened The new AI Router proxies to Dedicated inference deployments through mTLS. Some of the mTLS resources were missing for some deployments. Because of this misconfiguration, impacted deployments returned HTTP 502 errors for all inference requests. What We Learned We didn’t have a proper way to validate that all deployments had their mTLS configuration set up before the maintenance, and we incorrectly assumed this was true for all of them. What We’ve Changed We took an inventory and created the missing resources. Moving Forward Future infrastructural rollouts will be performed more gradually, progressing deployment by deployment, staged per zone. Also, we are adding more processes to ensure the invariants that should hold are true before performing this kind of maintenance.
Your product runs on more than Exoscale
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services