Is Scalr down?

We are currently investigating this issue.

We are investigating an issue causing runs to fail with the error: “Cannot provision execution environment.” This does not impact runs on self-hosted agents.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

## Summary On October 22, 2026, between 15:45 and 16:35 UTC, some runs failed to start. Affected tasks failed with the error `“Cannot provision execution environment”`. The issue was caused by a sudden spike in run volume that exceeded a capacity limit in our cluster networking. It resolved on its own once the spike subsided, and the service is operating normally. A fix that removes the underlying cause is rolling out. ## Customer impact Between 15:45 and 16:35 UTC, ~2,000 tasks failed across ~1,800 runs, affecting roughly 100 accounts. Affected runs failed with an internal error because the agent worker never picked up the task, so no work was started. Tasks that were already running were not affected. If any of your runs failed during this window, please re-run them. ## Timeline All times are UTC on October 22, 2026. | Time | Event | | --- | --- | | 15:45 | Runs begin failing to start as a surge of run activity pushes the platform past a networking capacity limit. | | 16:35 | The surge subsides, capacity recovers, and new runs start normally. | | After recovery | The cause is identified and a fix begins rolling out. | ## Root cause Each agent task runs in its own short-lived pod, and each pod was labeled with values unique to that task. Our cluster networking creates a separate network identity for every unique set of labels, so each task created its own identity. Those identities are tracked in a table with a fixed size limit. At roughly twice our normal hourly task volume, the number of identities exceeded that limit. Pods that started while the table was full came up without DNS access, so the agent worker could not pick up the task, and it failed after 300 seconds. The limit existed before this event. A burst of runs from several customers across many workspaces was what pushed us over it. ## Resolution The issue cleared on its own as the burst subsided and unused network identities were cleaned up. In parallel, we began rolling out a release that removes the per-task unique values from task pod labels, so tasks no longer each create their own identity. This reduces identity usage by orders of magnitude, well below the limit. ## Preventing a recurrence 1. **Remove the cause.** We are rolling out the label change described above, narrowing the DNS network rule on task pods so it consumes far less capacity, and further hardening our networking configuration, which we will validate outside production first. 2. **Improve monitoring.** We are adding alerts on networking capacity so we are warned well before the limit is reached. 3. **Detect failures sooner.** We are adding alerts on the rate of internal-error run failures and on the `Task was not started by the agent worker within 300s` error, so issues like this are caught quickly regardless of cause. ## Our apology We are sorry for the disruption this caused your teams and pipelines. Reliable run execution is core to what you depend on us for. If you have questions about how this affected your account, please contact Scalr support.

We are investigating reports of errors affecting Scalr customers. Customers may see timeouts or failed requests in their runs. We are looking into this and will post another update as soon as we have more information.

Google Cloud experienced network degradation across multiple services, which has since been resolved.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

Customers who force agent pool assignment on workspaces are seeing the error "Self-hosted agent pool assignment is required by policy." even when an agent is assigned. We are rolling back the code that caused this issue now.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Root cause** A regression in a release deployed the morning of 2026-08-25 caused the run policy engine to incorrectly flag self-hosted agent pool assignments as non-compliant, triggering the error above even for correctly configured workspaces. **Resolution** The release was rolled back, restoring normal behavior. No customer action is required. **Preventive measures** We are adding regression test coverage for this specific scenario \(self-hosted agent enforcement combined with valid pool assignment\) to prevent recurrence, and reviewing our release validation process for this code path. We apologize for the disruption this caused to your pipelines.

The docs site is currently down. We are working with the vendor to understand what the issue is.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.