Incident with Codespaces

Update timeline

1. investigating Mar 09, 2026, 03:04 AM UTC

   We are investigating reports of degraded performance for Codespaces

2. investigating Mar 09, 2026, 03:04 AM UTC

   We are seeing about 5% of new Codespace creation requests failing. We are investigating the root cause and identifying the impacted regions.

3. investigating Mar 09, 2026, 03:32 AM UTC

   We are seeing recovery, with the failure rate for new Codespace creation requests dropping from 5% to about 3%.

4. investigating Mar 09, 2026, 03:51 AM UTC

   This incident has been resolved. New Codespace creation requests are now completing successfully.

5. resolved Mar 09, 2026, 03:51 AM UTC

   On March 9, 2026, between 01:23 UTC and 03:25 UTC, users attempting to create or resume codespaces in the Australia East region experienced elevated failures, peaking at a 100% failure rate for this region. Codespaces in other regions were not affected. The create and resume failures were caused by degraded network connectivity between our control plane services and the VMs hosting the codespaces. This was resolved by redirecting traffic to an alternate site within the region. While we are addressing the core network infrastructure issue, we have also improved our observability of components in this area to improve detection. This will also enable our existing automated failovers to cover this failure mode. These changes will prevent or significantly reduce the time any similar incident causes user impact.