Scalr Outage History

Started Sep 01, 2026, 03:44 PM UTC· resolved Sep 01, 2026, 05:06 PM UTC

1. investigating Sep 01, 2026, 03:44 PM UTC

   We are investigating reports of errors affecting Scalr customers. Customers may see timeouts or failed requests in their runs. We are looking into this and will post another update as soon as we have more information.

2. monitoring Sep 01, 2026, 05:00 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 01, 2026, 05:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 27, 2026, 03:54 PM UTC· resolved Aug 27, 2026, 04:09 PM UTC

1. investigating Aug 27, 2026, 03:54 PM UTC

   We are currently investigating this issue.

2. resolved Aug 27, 2026, 04:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 25, 2026, 12:23 PM UTC· resolved Aug 25, 2026, 01:31 PM UTC

1. identified Aug 25, 2026, 12:23 PM UTC

   Customers who force agent pool assignment on workspaces are seeing the error "Self-hosted agent pool assignment is required by policy." even when an agent is assigned. We are rolling back the code that caused this issue now.

2. monitoring Aug 25, 2026, 01:11 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 25, 2026, 01:31 PM UTC

   This incident has been resolved.

4. postmortem Aug 25, 2026, 07:09 PM UTC

   **Root cause** A regression in a release deployed the morning of 2026-08-25 caused the run policy engine to incorrectly flag self-hosted agent pool assignments as non-compliant, triggering the error above even for correctly configured workspaces. **Resolution** The release was rolled back, restoring normal behavior. No customer action is required. **Preventive measures** We are adding regression test coverage for this specific scenario \(self-hosted agent enforcement combined with valid pool assignment\) to prevent recurrence, and reviewing our release validation process for this code path. We apologize for the disruption this caused to your pipelines.

Read the full incident report →

Started Aug 20, 2026, 04:46 PM UTC· resolved Aug 20, 2026, 04:49 PM UTC

1. investigating Aug 20, 2026, 04:46 PM UTC

   The docs site is currently down. We are working with the vendor to understand what the issue is.

2. monitoring Aug 20, 2026, 04:48 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 20, 2026, 04:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 06, 2026, 04:48 PM UTC· resolved Jul 06, 2026, 05:10 PM UTC

1. investigating Jul 06, 2026, 04:48 PM UTC

   We are currently investigating this issue.

2. identified Jul 06, 2026, 04:50 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 06, 2026, 04:51 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Jul 06, 2026, 04:53 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Jul 06, 2026, 05:10 PM UTC

   This incident has been resolved.

6. postmortem Jul 07, 2026, 06:21 PM UTC

   **Summary** On July 6, 2026, runs failed to start for approximately 22 minutes, returning the error "Cannot provision execution environment." The root cause was a stuck PVC \(persistent volume claim\) during a migration to new network storage for the Terraform/OpenTofu providers cache. **Impact** Between 16:48 and 16:53 UTC, new runs across affected workspaces failed to schedule with a "Cannot provision execution environment" error. **Timeline** \(all times UTC\) * 16:48 — Issue detected; investigation began * 16:50 — Root cause identified: PVC stuck due to in-use protection finalizer * 16:51 — Mitigation deployed \(providers cache disabled\); monitoring began * 16:53 — Continued monitoring, no further impact observed * 17:10 — Incident resolved **What Happened** During the migration to new network storage for the Terraform/OpenTofu providers cache, the existing PVC was still attached to multiple active workloads. Kubernetes' in-use protection finalizer blocked deletion and recreation of the volume, so new runs could not bind to it and failed to schedule. **Mitigation** We disabled the providers cache to unblock run scheduling while we worked on the underlying volume issue. This restored run scheduling immediately, with only a minor, temporary performance impact from running without a warm provider cache. **Root Cause** The migration runbook did not account for the PVC's in-use protection finalizer when workloads were still actively bound to the volume, which blocked the planned delete-and-recreate step. **Remediation and Prevention** * Immediate: delete the workloads still holding the PVC, recreate the volume, and restore it as the providers cache. * Preventive: update the migration runbook to explicitly drain or cordon any workloads still bound to a PVC before attempting to delete and recreate it, rather than assuming the volume is free.

Read the full incident report →

Started Jul 01, 2026, 06:30 PM UTC· resolved Jul 01, 2026, 09:46 PM UTC

1. investigating Jul 01, 2026, 06:40 PM UTC

   We are currently investigating this issue.

2. identified Jul 01, 2026, 06:47 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jul 01, 2026, 08:39 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jul 01, 2026, 08:42 PM UTC

   Customers should see the lagging runs start to improve. Existing runs will resume normally on their own.

5. monitoring Jul 01, 2026, 09:17 PM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Jul 01, 2026, 09:17 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Jul 01, 2026, 09:46 PM UTC

   This incident has been resolved.

8. postmortem Jul 02, 2026, 02:27 PM UTC

   **Summary** On July 1, Terraform and OpenTofu runs experienced significant delays, up to 10–25 minutes, during the initialization and plan phases for a number of customers, due to capacity constraints in the network-backed provider plugin cache under load. We have mitigated the issue by temporarily removing the shared network storage from the provider download path, and the platform is stable. We are re-architecting how provider plugins are distributed to prevent this class of slowdown from recurring. **What Happened** Before a run can begin planning, it downloads the Terraform/OpenTofu provider plugins it needs \(for example, the AWS, GCP, or Azure providers\) from a shared cache backed by network storage. A period of high run volume, combined with growth in the total size of the provider cache, drove the aggregate load past the practical limits of that storage. As downloads queued, the init/plan phase of affected runs slowed dramatically, in some cases by 10–25 minutes, and a small number of runs errored or hit their timeout and had to be retried. The network storage reached its limits due to the large number of connected run nodes and began throttling, slowing down all workloads that relied on it. **Mitigation** After several unsuccessful attempts to increase the storage throughput, we temporarily removed the shared network storage from the provider cache path, providers are currently downloaded over the public network for each run. As a result, init times may be slightly longer than with a healthy provider cache, and runs temporarily depend on the availability of public provider registries. **What We're Doing Next** We are improving the plugin cache architecture along two lines: a node-local caching layer that eliminates the shared network storage bottleneck \(to be in place before we re-enable the provider cache\), and a caching network mirror that removes the direct dependency on public registry availability. Together, these layers ensure the failure of any one of them is absorbed by the others without service degradation. We apologize for the disruption. If you have questions or are still experiencing issues, please contact our support team.

Read the full incident report →

Started May 21, 2026, 04:24 PM UTC· resolved May 21, 2026, 06:44 PM UTC

1. investigating May 21, 2026, 04:24 PM UTC

   Some accounts are experiencing longer than usual wait times for runs to execute.

2. identified May 21, 2026, 04:24 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 21, 2026, 06:27 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 21, 2026, 06:44 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 19, 2026, 10:17 AM UTC· resolved May 19, 2026, 11:08 AM UTC

1. investigating May 19, 2026, 10:17 AM UTC

   Some customers are unable to log into Scalr due to SSL errors.

2. identified May 19, 2026, 10:17 AM UTC

   The issue has been identified and a fix is being implemented.

3. identified May 19, 2026, 10:18 AM UTC

   The release is being rolled back and we expect the fix to be completed in ~10-15 minutes

4. monitoring May 19, 2026, 11:01 AM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring May 19, 2026, 11:01 AM UTC

   We are continuing to monitor for any further issues.

6. resolved May 19, 2026, 11:08 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 18, 2026, 09:29 AM UTC· resolved May 18, 2026, 10:26 AM UTC

1. investigating May 18, 2026, 09:29 AM UTC

   We're experiencing an elevated level of run issues and are currently looking into the issue.

2. monitoring May 18, 2026, 10:21 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 18, 2026, 10:26 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 06, 2026, 03:31 PM UTC· resolved May 06, 2026, 04:48 PM UTC

1. investigating May 06, 2026, 03:31 PM UTC

   Some accounts are experiencing delays with runs starting.

2. identified May 06, 2026, 03:43 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 06, 2026, 04:24 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring May 06, 2026, 04:36 PM UTC

   We are continuing to monitor for any further issues.

5. resolved May 06, 2026, 04:48 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 29, 2026, 03:22 PM UTC· resolved Apr 29, 2026, 06:57 PM UTC

1. investigating Apr 29, 2026, 03:22 PM UTC

   We aare currently investigating an issue where some runs are stuck in a pending approval state without the ability to approve them.

2. identified Apr 29, 2026, 03:22 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 29, 2026, 05:04 PM UTC

   A fix has been implemented and we are monitoring the results.

4. identified Apr 29, 2026, 05:08 PM UTC

   The issue has been identified and a fix is being implemented.

5. monitoring Apr 29, 2026, 05:11 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Apr 29, 2026, 06:57 PM UTC

   This incident has been resolved.

7. postmortem Apr 30, 2026, 06:03 PM UTC

   **Update #1** On April 28 at approximately 15:00 UTC and again on April 29 at approximately 15:22 UTC, a load spike affected our internal task queue, which slowed down the mechanism Scalr uses to transition runs between pipeline stages. In some cases, the transition task failed before it could complete, leaving runs stuck in a waiting state with no automatic recovery path. Both times, the queues cleared on their own while our engineering team investigated. We have already shipped an improvement \(released April 30\) that makes this transition task significantly more resilient to lock contention, greatly reducing the likelihood of runs getting stuck. We are continuing to investigate the underlying cause of the queue spikes and will share further updates as the investigation progresses.

Read the full incident report →

Started Apr 28, 2026, 03:00 PM UTC· resolved Apr 28, 2026, 03:45 PM UTC

1. investigating Apr 28, 2026, 03:13 PM UTC

   We are currently investigating this issue.

2. identified Apr 28, 2026, 03:34 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Apr 28, 2026, 03:45 PM UTC

   This incident has been resolved.

4. postmortem Apr 30, 2026, 06:11 PM UTC

   Please follow updates [here](https://status.scalr.io/incidents/mjzqwnbjnt9f).

Read the full incident report →

Started Apr 28, 2026, 08:53 AM UTC· resolved Apr 28, 2026, 09:52 AM UTC

1. investigating Apr 28, 2026, 08:53 AM UTC

   We are currently investigating this issue.

2. investigating Apr 28, 2026, 09:00 AM UTC

   We are continuing to investigate this issue.

3. investigating Apr 28, 2026, 09:07 AM UTC

   We are continuing to investigate this issue.

4. investigating Apr 28, 2026, 09:08 AM UTC

   We are continuing to investigate this issue.

5. identified Apr 28, 2026, 09:18 AM UTC

   The issue has been identified and a fix is being implemented.

6. monitoring Apr 28, 2026, 09:21 AM UTC

   A fix has been implemented and we are monitoring the results.

7. resolved Apr 28, 2026, 09:52 AM UTC

   This incident has been resolved.

8. postmortem May 01, 2026, 02:38 PM UTC

   **Summary** A backend optimization deployed on April 24 inadvertently triggered a database performance issue that caused a platform-wide outage. The change has been fully reverted and the platform is stable. We are addressing the underlying query design before re-attempting the optimization. **What Happened** The optimization changed how policy code is delivered during runs, moving from an inline payload to a download from blob storage. This introduced an additional authorization check on each policy download that was not present before. That authorization check relied on an existing database query with significant hidden complexity: under normal conditions it goes unnoticed, but at production policy-check volume it joined approximately 2 million rows per call, taking around 25 seconds to complete. The increased call frequency exposed this latency, drove the database to 100% CPU utilization, and exhausted the connection pool, making the platform unavailable. **Resolution** The optimization was fully reverted, restoring normal platform behavior. No data was lost or corrupted. **What We're Doing Next** The authorization query is being redesigned to use a direct indexed lookup, which eliminates the row-scan behavior that caused the spike. The optimization will not be re-released until this redesign is complete and validated. We apologize for the disruption. If you have questions or are still experiencing issues, please contact our support team.

Read the full incident report →

Started Apr 14, 2026, 08:59 AM UTC· resolved Apr 14, 2026, 11:19 AM UTC

1. investigating Apr 14, 2026, 08:59 AM UTC

   We are currently investigating this issue.

2. identified Apr 14, 2026, 09:14 AM UTC

   We've identified the root cause and are actively working on a fix.

3. identified Apr 14, 2026, 10:09 AM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Apr 14, 2026, 11:12 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Apr 14, 2026, 11:19 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 16, 2025, 10:13 AM UTC· resolved Dec 16, 2025, 01:07 PM UTC

1. investigating Dec 16, 2025, 10:13 AM UTC

   Currently we are experiencing a CDN issue with the Scalr UI.

2. identified Dec 16, 2025, 10:15 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Dec 16, 2025, 12:27 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 16, 2025, 01:07 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 24, 2025, 04:25 PM UTC· resolved Oct 24, 2025, 07:13 PM UTC

1. investigating Oct 24, 2025, 04:25 PM UTC

   We are currently investigating this issue.

2. identified Oct 24, 2025, 04:28 PM UTC

   We've implemented a hotfix that should significantly decrease number of errors.

3. monitoring Oct 24, 2025, 04:37 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Oct 24, 2025, 05:11 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Oct 24, 2025, 07:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 07, 2025, 08:42 PM UTC· resolved Oct 07, 2025, 10:14 PM UTC

1. investigating Oct 07, 2025, 08:42 PM UTC

   Runs are not using agents if the agent is set as the default for the account. A workaround, for now, is to set the agent pool in the workspace settings explicitly. A fix is in progress to ensure that the account-level defaults are used as expected.

2. identified Oct 07, 2025, 08:42 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Oct 07, 2025, 10:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 25, 2025, 12:57 AM UTC· resolved Sep 25, 2025, 01:30 AM UTC

1. investigating Sep 25, 2025, 12:57 AM UTC

   The Docker Registry outage is impacting Scalr's ability to pull images, resulting in run failures. We are actively monitoring the status at https://www.dockerstatus.com/

2. identified Sep 25, 2025, 12:57 AM UTC

   From Docker: We are continuing to work on implementing a fix. We will update as the status evolves. https://www.dockerstatus.com/

3. resolved Sep 25, 2025, 01:30 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 02, 2025, 11:07 AM UTC· resolved Sep 02, 2025, 12:21 PM UTC

1. investigating Sep 02, 2025, 11:07 AM UTC

   We're currently investigating an issue where Azure VCS integration cannot be created or reauthorised with Scalr-managed Azure application:"AADSTS700016: Application with identifier 'c375f129-2031-43cf-8e01-a8cdac263bf1' was not found in the directory 'Default Directory'". If you encounter this issue, please temporarily use another authentication method such as a Custom Azure Application or a Personal Access Token.

2. monitoring Sep 02, 2025, 11:41 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 02, 2025, 12:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 21, 2025, 08:06 PM UTC· resolved Aug 21, 2025, 08:23 PM UTC

1. investigating Aug 21, 2025, 08:06 PM UTC

   We're currently investigating an issue where GitLab integration can stop working with an error: "The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client." If you experience this issue, please manually re-authorize GitLab connection from the Integrations page. Contact support if you have further questions.

2. resolved Aug 21, 2025, 08:23 PM UTC

   Issue is related to a GitLab incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20398 . If your integration is affected, please re-authorize GitLab integration to fix it.

Read the full incident report →

Started Aug 18, 2025, 02:06 PM UTC· resolved Aug 18, 2025, 04:09 PM UTC

1. investigating Aug 18, 2025, 02:06 PM UTC

   The scalr.com website is down due to an issue with Cloudflare. This does not impact the scalr.io or the Scalr application. Users can continue to sign up at scalr.io

2. resolved Aug 18, 2025, 04:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 13, 2025, 11:00 AM UTC· resolved Aug 13, 2025, 11:15 PM UTC

1. investigating Aug 13, 2025, 11:00 AM UTC

   Some users are experiencing intermittent connection issues with the Azure provider.

2. investigating Aug 13, 2025, 09:04 PM UTC

   We are continuing to investigate the intermittent connection issues between Google Cloud and Azure. This is impacting a few customers. Please get in touch with us for further options if this is impacting your ability to execute runs.

3. monitoring Aug 13, 2025, 11:05 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 14, 2025, 08:23 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 12, 2025, 06:22 PM UTC· resolved Jun 12, 2025, 09:17 PM UTC

1. investigating Jun 12, 2025, 06:22 PM UTC

   We are currently investigating this issue.

2. investigating Jun 12, 2025, 06:38 PM UTC

   Google cloud storage is currently experiencing issues.

3. investigating Jun 12, 2025, 06:49 PM UTC

   Google Cloud storage is experiencing a widespread outage and not accessible.

4. investigating Jun 12, 2025, 06:49 PM UTC

   We are continuing to investigate this issue.

5. identified Jun 12, 2025, 07:22 PM UTC

   Google Cloud has identified and is mitigating the issue. They have confirmed the issue has recovered in some locations.

6. identified Jun 12, 2025, 07:22 PM UTC

   Google Cloud has identified and is mitigating the issue. They have confirmed the issue has recovered in some locations. Scalr is still seeing intermittent issues.

7. monitoring Jun 12, 2025, 08:43 PM UTC

   According to our monitoring, less than 1% of requests are currently affected by the Google incident. Majority of the users should see a full recovery at this point.

8. resolved Jun 12, 2025, 09:17 PM UTC

   This incident has been resolved.

9. postmortem Jun 17, 2025, 01:10 PM UTC

   On Thursday, June 12, 2025, starting at approximately 2:00 PM EST, our customers experienced service errors and degraded performance. Full service was restored for all users by 5:15 PM EST. This disruption was caused by a major global outage across Google Cloud services, which our platform relies on. According to Google's post-incident analysis, a recent update to their internal "Service Control" system contained a bug. This bug was triggered by a routine policy change, causing a critical component of their API infrastructure to fail worldwide. This failure resulted in the errors our customers experienced. The detailed RCA can be found here: [https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) We apologize for the significant impact this had on your operations and appreciate your understanding. The Scalr Team

Read the full incident report →

Started May 13, 2025, 08:05 PM UTC· resolved May 14, 2025, 01:24 PM UTC

1. investigating May 13, 2025, 08:05 PM UTC

   We want to inform you of a recent issue affecting our auto-apply functionality. We've identified a bug where auto-apply is automatically disabled when the latest state comes from a branch that differs from your workspace branch. The bug was introduced today, and our engineering team is actively working on a fix that will be deployed within the next 12-24 hours. We wanted to make you aware in the event that you see a workspace that should be auto-applied, but is waiting on approval. We apologize for the inconvenience, and please let us know if you have any questions by opening a ticket at support.scalr.com

2. identified May 13, 2025, 08:06 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 14, 2025, 01:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 05, 2025, 03:30 PM UTC· resolved May 05, 2025, 03:30 PM UTC

1. resolved May 07, 2025, 01:21 PM UTC

   We're currently working on a fix for runs that are failing with: │ Error: Failed to parse command-line flags │ │ flag provided but not defined: -var-file

2. postmortem May 07, 2025, 01:22 PM UTC

   **RCA** A fix was released to ensure `.tfvars` files are properly loaded during the `init` phase with `for-each` feature, that was introduced in the OpenTofu [1.9.0](https://opentofu.org/blog/opentofu-1-9-0/) version. Technical details: * To fix the issue we added processing of the `.tfvars` on `init`, `validate, `and `fmt` phases for runs with the flag `-var-file.` * But consuming the `.tfvars` file using the `-var` and `-var-file` options was introduced in OpenTofu [1.8.0](https://opentofu.org/docs/v1.8/intro/whats-new/), and exception handling was missed for any versions less than 1.8.0. * Because of this, all runs with `.tfvars` files in the VCS-driven workspaces with OpenTofu < 1.8.0 and all Terraform versions supported by Scalr were affected.

Read the full incident report →