[Australia] [Serverless] startup delays for Serverless Jobs

Update timeline

1. identified Jul 17, 2026, 09:24 PM UTC

   The issue causing degraded performance impacting Job execution for Serverless in Australia has been identified, and corrective actions are underway. Impact: Users may continue to see delays when starting Jobs. We will provide further updates as mitigation progresses.

2. monitoring Jul 17, 2026, 11:00 PM UTC

   The issue has been mitigated. We are continuing to monitor job execution.

3. resolved Jul 17, 2026, 11:27 PM UTC

   The issue has been resolved, and job execution is functioning normally.

4. postmortem Jul 27, 2026, 07:14 PM UTC

   ## Customer impact Between July 17, 2026 at 9:18 pm UTC and July 17, 2026 at 11:27 pm UTC, a subset of customers in Australia experienced delays when starting serverless jobs. Jobs were expected to fall back to an alternate execution path, but that fallback added startup latency while the platform detected the failed start and retried execution. The incident lasted 2 hours and 8 minutes, with mitigation reached at July 17, 2026 at 11:00 pm UTC and full recovery confirmed at July 17, 2026 at 11:27 pm UTC. ## Root cause Our cloud service provider introduced new CPU hardware \(a new processor model\) into the virtual machine family used to run serverless job execution in Australia. The virtualization runtime we use to isolate job workloads was incompatible with the new hardware, so job execution environments did not start successfully on newly provisioned capacity. Because those environments could not start on the affected resources, serverless jobs were retried through a fallback execution path. That fallback helped jobs continue running, but it introduced delays while the platform recognized the startup failure and redirected execution. The same issue had been observed in the United States region the previous day, which enabled the team to identify the cause quickly and deploy a fix that had already been developed. ## Detection The issue was detected through automated alerting for the serverless job execution service on July 17, 2026 at 9:18 pm UTC. Our engineering team began investigating immediately. The issue was confirmed as customer-impacting on July 17, 2026 at 9:19 pm UTC. ## Response On July 17, 2026 at 9:24 pm UTC, we published a customer status update confirming that the cause of degraded serverless job execution performance had been identified and that corrective actions were underway. As an immediate step, the team removed the affected compute resources to attempt to obtain compatible replacements. In parallel, our engineering team deployed a fix to address the compute resource incompatibility. By July 17, 2026 at 10:04 pm UTC, the fix was being deployed and was expected to complete shortly. After the fix was applied and verified, we moved the incident to monitoring on July 17, 2026 at 11:00 pm UTC and continued validating job execution. On July 17, 2026 at 11:27 pm UTC, we confirmed that job execution was functioning normally and marked the incident resolved. ## Follow-up 1. Deploy a validation component that verifies newly provisioned compute capacity can run job workloads before customer jobs are scheduled onto it. \(Completed\) 2. Update the job isolation runtime so workloads start correctly on the new processor generation, removing the incompatibility. \(In progress\) 3. Engage the cloud service provider regarding the introduction of new CPU hardware into the requested virtual machine family, to improve advance visibility of hardware changes that can affect workload compatibility.