Scalr experienced a notice incident on July 1, 2026 affecting Scalr Platform and Scalr Worker, lasting 3h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 01, 2026, 06:40 PM UTC
We are currently investigating this issue.
- identified Jul 01, 2026, 06:47 PM UTC
The issue has been identified and a fix is being implemented.
- identified Jul 01, 2026, 08:39 PM UTC
We are continuing to work on a fix for this issue.
- identified Jul 01, 2026, 08:42 PM UTC
Customers should see the lagging runs start to improve. Existing runs will resume normally on their own.
- monitoring Jul 01, 2026, 09:17 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jul 01, 2026, 09:17 PM UTC
We are continuing to monitor for any further issues.
- resolved Jul 01, 2026, 09:46 PM UTC
This incident has been resolved.
- postmortem Jul 02, 2026, 02:27 PM UTC
**Summary** On July 1, Terraform and OpenTofu runs experienced significant delays, up to 10–25 minutes, during the initialization and plan phases for a number of customers, due to capacity constraints in the network-backed provider plugin cache under load. We have mitigated the issue by temporarily removing the shared network storage from the provider download path, and the platform is stable. We are re-architecting how provider plugins are distributed to prevent this class of slowdown from recurring. **What Happened** Before a run can begin planning, it downloads the Terraform/OpenTofu provider plugins it needs \(for example, the AWS, GCP, or Azure providers\) from a shared cache backed by network storage. A period of high run volume, combined with growth in the total size of the provider cache, drove the aggregate load past the practical limits of that storage. As downloads queued, the init/plan phase of affected runs slowed dramatically, in some cases by 10–25 minutes, and a small number of runs errored or hit their timeout and had to be retried. The network storage reached its limits due to the large number of connected run nodes and began throttling, slowing down all workloads that relied on it. **Mitigation** After several unsuccessful attempts to increase the storage throughput, we temporarily removed the shared network storage from the provider cache path, providers are currently downloaded over the public network for each run. As a result, init times may be slightly longer than with a healthy provider cache, and runs temporarily depend on the availability of public provider registries. **What We're Doing Next** We are improving the plugin cache architecture along two lines: a node-local caching layer that eliminates the shared network storage bottleneck \(to be in place before we re-enable the provider cache\), and a caching network mirror that removes the direct dependency on public registry availability. Together, these layers ensure the failure of any one of them is absorbed by the others without service degradation. We apologize for the disruption. If you have questions or are still experiencing issues, please contact our support team.