Spacelift Outage History
Spacelift is up right nowPingoru has recorded 52 incidents for Spacelift: 14 in the last 12 months and 4 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 24 of 720 (3%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 1 | 2 of 720 (0%) |
| July 2026 | 2 | 74 of 744 (10%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 0 of 720 (0%) |
| October 2026 | 2 | 3 of 510 (1%) |
Incidents, newest first
Major October 21, 2026 US Customers MQTT Service Disruption 1h 22m Public workers
Started Oct 21, 2026, 12:03 AM UTC· resolved Oct 21, 2026, 01:26 AM UTC
- investigating Oct 21, 2026, 12:03 AM UTC
We’re seeing an increased number of errors affecting the worker polling API for customers in our US region. Our team is actively working on mitigation and will share updates as they become available.
- identified Oct 21, 2026, 12:17 AM UTC
We have the fix implemented and we will release it shortly.
- monitoring Oct 21, 2026, 01:10 AM UTC
We deployed the fix and monitoring.
- resolved Oct 21, 2026, 01:26 AM UTC
This incident has been resolved.
Major October 20, 2026 Polling Workers Degragation 1h 31m Public workers
Started Oct 20, 2026, 02:05 PM UTC· resolved Oct 20, 2026, 03:36 PM UTC
- identified Oct 20, 2026, 02:05 PM UTC
We have noticed increased number of errors related to polling workers API, we are mitigating the problem.
- identified Oct 20, 2026, 02:49 PM UTC
We have the fix implemented and we will release it shortly.
- resolved Oct 20, 2026, 03:36 PM UTC
Root cause: A software defect caused the worker communication service in the EU region to crash and restart repeatedly. Impacted customers: Customers whose workers use the poll-based communication mode. Runs on those workers were stuck when the run was triggered manually by a user and the stack had a server-side initialization policy attached. Other polling workers may have been briefly delayed while the service restarted.
Major September 11, 2026 Event Processing Delays 20m Event processing
Started Sep 11, 2026, 01:22 PM UTC· resolved Sep 11, 2026, 01:43 PM UTC
- identified Sep 11, 2026, 01:22 PM UTC
We're seeing increased delays and/or elevated error rates in our FedRamp event processing queue. The root cause has been identified, we working on restoring the regular functionality of the system.
- resolved Sep 11, 2026, 01:43 PM UTC
We have restored events processing to the usual state. The root cause was a change that we released earlier today.
Notice July 29, 2026 AWS connectivity for runs using hashicorp/aws v6.57.0 2h 5m
Started Jul 29, 2026, 01:39 PM UTC· resolved Jul 29, 2026, 03:44 PM UTC
- investigating Jul 29, 2026, 01:39 PM UTC
We are experiencing failing AWS API calls for runs using hashicorp/aws provider with version v6.57.0.
- identified Jul 29, 2026, 03:22 PM UTC
We have identified a problem with v6.57.0 of the hashicorp/aws provider causing requests for certain resources to fail. A workaround for affected customers is to pin to v6.56.0 or below. More details can be found on the HashiCorp status page: https://status.hashicorp.com/incidents/01KYPY1KTBE8KWC4E75RXN0XX8
- resolved Jul 29, 2026, 03:44 PM UTC
This incident has been resolved.
Notice July 13, 2026 External symlink configuration issue 2d 23h
Started Jul 13, 2026, 03:19 PM UTC· resolved Jul 16, 2026, 02:52 PM UTC
- identified Jul 13, 2026, 03:19 PM UTC
We have identified an issue with external symlink configurations affecting some setups. Please contact Support if you are affected.
- resolved Jul 16, 2026, 02:52 PM UTC
The incident has been resolved.
Major June 1, 2026 Runs failing on workers 2h 17m Public workers
Started Jun 01, 2026, 08:51 AM UTC· resolved Jun 01, 2026, 11:09 AM UTC
- investigating Jun 01, 2026, 08:51 AM UTC
We've spotted as issue with failing runs. We're currently investigating the issue, and will provide an update soon.
- identified Jun 01, 2026, 09:22 AM UTC
The issue has been identified. This is not affecting public workers; only private workers can be affected. The last working version of the worker's image is [v1.79.0](https://github.com/spacelift-io/spacelift-worker-image/releases/tag/1.79.0). Please rollback to that version if you see failing runs after an update.
- identified Jun 01, 2026, 10:11 AM UTC
As a mitigation we have deregistered all of the AMIs for version 1.80.0 and 1.81.0. We are currently working on building new images that include a fix for the problem.
- resolved Jun 01, 2026, 11:09 AM UTC
This incident has been resolved. A [new version](https://github.com/spacelift-io/spacelift-worker-image/releases/tag/1.82.0) of the image has been released.
Major April 20, 2026 Terraform runs failing due to expired HashiCorp signing key 21h 36m
Started Apr 20, 2026, 11:43 AM UTC· resolved Apr 21, 2026, 09:20 AM UTC
- monitoring Apr 20, 2026, 11:43 AM UTC
We identified that Terraform runs were failing when downloading the Terraform binary. The public GPG key we use to verify HashiCorp's checksum signatures had expired, causing every binary download to fail signature verification. Runs using OpenTofu or a Custom workflow were not affected. A fix updating the embedded HashiCorp public key has been rolled out during the week-end across our managed infrastructure. If you are using private worker pools, you will need to cycle your worker pool so the workers pick up the latest launcher version that includes the fix. Until workers are cycled, new Terraform runs on those pools will continue to fail. Action required for Self-Hosted: Hotfix v5.1.1 has been published and can be used to resolve the issue on Self-Hosted installations on v5.1.0. If you are on an older Self-Hosted version, please contact your Spacelift account team — we are publishing targeted hotfixes on request. If you cannot upgrade right now, you can unblock Terraform runs by: - Switch from Terraform to OpenTofu if you can. - Switch to a custom workflow for your stack and provide Terraform binary yourself - Setting "skipVendorBinaryDownload: true" on the affected stacks, and running those stacks in a runner image that already has the terraform binary installed at the expected path.
- monitoring Apr 21, 2026, 08:32 AM UTC
We are continuing to monitor for any further issues.
- resolved Apr 21, 2026, 09:20 AM UTC
If self hosted customer are still affected, please reach out to our support team.
Major April 17, 2026 Git clone operations failing with VCS agents 1h 17m
Started Apr 17, 2026, 01:45 PM UTC· resolved Apr 17, 2026, 03:02 PM UTC
- identified Apr 17, 2026, 01:45 PM UTC
A third party library update broke git operations when using a VCS agent. We are rolling out to the previous version.
- monitoring Apr 17, 2026, 02:25 PM UTC
A fix has been deployed to make sure git operations using VCS agents will operate correctly from now. We are still monitoring errors.
- resolved Apr 17, 2026, 03:02 PM UTC
This incident has been resolved.
Major April 14, 2026 Issues resolving correct module configuration when creating a new module version 48m Event processing
Started Apr 14, 2026, 01:35 PM UTC· resolved Apr 14, 2026, 02:23 PM UTC
- investigating Apr 14, 2026, 01:35 PM UTC
We're currently investigating the issue, and will provide an update soon.
- identified Apr 14, 2026, 02:04 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 14, 2026, 02:08 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 14, 2026, 02:23 PM UTC
This incident has been resolved.
Minor March 31, 2026 Spacelift Terraform Provider v1.47.0 incorrect signature 20m
Started Mar 31, 2026, 02:25 PM UTC· resolved Mar 31, 2026, 02:45 PM UTC
- investigating Mar 31, 2026, 02:25 PM UTC
We've spotted that Spacelift Terraform Provider version v1.47.0 has incorrect signature. We're currently investigating the issue, and will provide an update soon.
- identified Mar 31, 2026, 02:28 PM UTC
We have identified the root cause to be in the goreleaser: https://github.com/goreleaser/goreleaser/issues/6514
- identified Mar 31, 2026, 02:32 PM UTC
We have pinned goreleaser to an older version, and we are releasing an updated version of our provider (v1.47.1) with a correct signature.
- monitoring Mar 31, 2026, 02:42 PM UTC
Version v1.47.1 released. We are waiting for registries to pick up the new version, then we will confirm the fix.
- resolved Mar 31, 2026, 02:45 PM UTC
We have confirmed the correct functionality of v1.47.1
Major March 16, 2026 Environment disruptions (app.us.spacelift.io) 3h 39m
Started Mar 16, 2026, 04:24 PM UTC· resolved Mar 16, 2026, 08:03 PM UTC
- monitoring Mar 16, 2026, 04:24 PM UTC
We identified that an infrastructure change resulting in disruptions for US accounts, such as workers disconnecting from Spacelift and webhook-triggered runs not being processed. The main issue has been resolved, and message processing has recovered. Only US region customers were impacted.
- monitoring Mar 16, 2026, 05:09 PM UTC
We identified an infrastructure change that caused disruptions for US accounts, including workers disconnecting from Spacelift and webhook-triggered runs not being processed. The main issue has been resolved, and message processing has recovered. Only customers in the US region were impacted.
- monitoring Mar 16, 2026, 05:51 PM UTC
We are still facing ongoing issues with stuck runs in our US environment and are working on automatic remediation.
- resolved Mar 16, 2026, 08:03 PM UTC
The issue has been resolved
Minor March 2, 2026 Trivy plugin failing with 404 error 1h 4m
Started Mar 02, 2026, 12:59 PM UTC· resolved Mar 02, 2026, 02:03 PM UTC
- identified Mar 02, 2026, 12:59 PM UTC
The Trivy Spacelift plugin is currently causing runs to fail during the Initializing phase due to a security incident from Trivy (https://github.com/aquasecurity/trivy/discussions/10265) that caused GitHub release artifacts to be deleted. We have identified the issue and are currently working on deploying a fix.
- resolved Mar 02, 2026, 02:03 PM UTC
This incident has been resolved. See the postmortem for resolution instructions.
- postmortem Mar 02, 2026, 02:53 PM UTC
📣 Trivy Plugin Action Required If you're using the previous version of the Trivy plugin, runs may fail during initialization due to missing release artifacts following a security incident in the upstream Trivy project \(details here: [https://github.com/aquasecurity/trivy/discussions/10265](https://github.com/aquasecurity/trivy/discussions/10265)\). We've updated the Spacelift Trivy plugin to **v0.69.2** and rolled the fix out to production - everything is now working as expected. To make sure your stacks use the updated version: 1. Go to **Plugins → Account plugins** and delete the existing Trivy plugin 2. Go to **Plugins → Templates** and install the Trivy plugin again Once done, all new runs will automatically use the fixed version. Thanks for your patience, and as always, reach out to support if you need anything.
Major November 18, 2025 Elevated GitHub Error Rate 48m HTTP API, API Errors
Started Nov 18, 2025, 09:21 PM UTC· resolved Nov 18, 2025, 10:10 PM UTC
- monitoring Nov 18, 2025, 09:21 PM UTC
We are experiencing elevated error rates from API requests made to GitHub and are monitoring the situation.
- resolved Nov 18, 2025, 10:10 PM UTC
This incident has been resolved.
Major October 17, 2025 Elevated API Errors 11m GraphQL API, Event processing, Platform UI
Started Oct 17, 2025, 09:36 AM UTC· resolved Oct 17, 2025, 09:47 AM UTC
- monitoring Oct 17, 2025, 10:04 AM UTC
We deployed a change that caused database-related issues. This resulted in a 11 minute outage. The problem was automatically resolved by our deployment process.
- resolved Oct 17, 2025, 10:05 AM UTC
This incident has been resolved.
Critical August 7, 2025 Spacelift unavailable 8m GraphQL API, Event processing
Started Aug 07, 2025, 01:23 PM UTC· resolved Aug 07, 2025, 01:31 PM UTC
- investigating Aug 07, 2025, 01:23 PM UTC
A recent deployment has caused Spacelift to become unavailable. We are working on rolling back to a working version and will post an update as soon as that has been done.
- resolved Aug 07, 2025, 01:31 PM UTC
We have successfully resolved the deployment issue and Spacelift is available again.
Major July 23, 2025 Incident affecting run scheduling 1h 37m Event processing, Public workers
Started Jul 23, 2025, 08:39 AM UTC· resolved Jul 23, 2025, 10:16 AM UTC
- investigating Jul 23, 2025, 08:39 AM UTC
We are currently investigating an incident affecting run scheduling and other asynchronous processing in Spacelift. The incident is causing delays with starting runs and updating statuses in VCS systems. As soon as we have more information we will post a further update.
- identified Jul 23, 2025, 08:49 AM UTC
The problem was caused by a large backlog of messages building up on one of our message queues. The backlog has cleared and the system appears to be operating correctly again. We are currently trying to understand the root cause to prevent it happening again.
- monitoring Jul 23, 2025, 09:40 AM UTC
We have identified the root cause of the incident. We are currently monitoring to make sure the system is stable, and we will take steps to prevent it occurring again.
- resolved Jul 23, 2025, 10:16 AM UTC
After continuing to monitor the system we have confirmed that it is stable again.
Notice June 27, 2025 Invalid CANCELED Value in Stack List 1h 27m
Started Jun 27, 2025, 12:46 PM UTC· resolved Jun 27, 2025, 02:13 PM UTC
- identified Jun 27, 2025, 12:46 PM UTC
We've found the cause of stack list errors affecting a small number of customers. The issue relates to canceled runs on a stack. A fix is in progress.
- monitoring Jun 27, 2025, 01:58 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 27, 2025, 02:13 PM UTC
This incident has been resolved.
Notice June 9, 2025 Degradation of the GraphQL API 1d
Started Jun 09, 2025, 02:44 AM UTC· resolved Jun 10, 2025, 03:05 AM UTC
- resolved Jun 11, 2025, 03:41 PM UTC
Between 02:44 and 03:05 our GraphQL API experienced an increased latency and a high error rate, causing most requests to fail. This was ultimately due to an issue with the autovacuum process of our Postgres database. More details are available in the post-mortem.
- postmortem Jun 11, 2025, 03:42 PM UTC
**Summary** Between June 8 and June 9, 2025, multiple service disruptions occurred due to increased latency and request failures across HTTP APIs. These incidents were traced back to the behavior of the PostgreSQL autovacuum process and were addressed by operational mitigations and subsequent investigations. **Incident Timeline** June 8, 2025: * 05:18 – 05:38 UTC: Service degradation. * 21:20 – 21:40 UTC: Another period of degradation. June 9, 2025: * 02:44 – 03:05 UTC: Additional disruption. * Post-incident: Investigation and fix deployed. **Root Cause** An unexpected PostgreSQL **autovacuum behavior** \(specifically, autovacuum acquiring an **ACCESS EXCLUSIVE** lock, which is highly unusual\) caused a series of latency spikes and request failures across our system. The trigger was a **data retention** job rolled out in early May 2025, which deleted large volumes of old data from a key table responsible for authorization logic. At first, the system appeared stable. However, **30 days after the initial clean-up job was completed**, PostgreSQL’s autovacuum initiated heap truncation—an optimization that reclaims empty pages at the end of a table. This delay occurred because pages only become truncatable once they become empty, and we have a 30-day retention on the data. Crucially, heap truncation requires an ACCESS EXCLUSIVE lock, the strictest type, which blocks all reads and writes to the table. This behavior was unexpected—autovacuum is typically non-blocking. **By default, PostgreSQL enables heap truncation**, but it rarely occurs and is generally harmless. It only becomes problematic on very busy tables, like ours, where even a brief exclusive lock can significantly disrupt live traffic. **Mitigations** * **Immediate**: Failovers were manually initiated to recover service during each incident. * **Permanent**: Heap truncation has been disabled on the affected table, and we are re-evaluating our data retention approach for tables like this to avoid future locking risks during maintenance operations. **Conclusion** These incidents were caused by an unintended interaction between large-scale data retention and PostgreSQL's autovacuum behavior on critical tables. Remediation steps are now in place to prevent recurrence, and process adjustments are underway to better assess such risks before future retention rollouts.
Major May 27, 2025 Spacelift GraphQL API unavailable 46m GraphQL API
Started May 27, 2025, 09:18 AM UTC· resolved May 27, 2025, 10:05 AM UTC
- identified May 27, 2025, 09:18 AM UTC
We've identified an issue with our latest production deployment that is causing Spacelift to be unavailable. We are working on rolling out a fix and will update this page as soon as we have more information.
- monitoring May 27, 2025, 09:38 AM UTC
We have put in place a fix and are now monitoring it to make sure that the problem is fully resolved.
- resolved May 27, 2025, 10:05 AM UTC
We have not detected any issues in the past 30 minutes and are marking this incident as resolved. We will investigate exactly how this incident occurred to try to prevent a similar issue in future.
Minor April 10, 2025 Slow database queries 1h 18m GraphQL API, Event processing
Started Apr 10, 2025, 04:37 AM UTC· resolved Apr 10, 2025, 05:56 AM UTC
- investigating Apr 10, 2025, 04:37 AM UTC
We are experiencing degradation in our database response times.
- identified Apr 10, 2025, 04:54 AM UTC
We've identified the core issue and currently are solving it.
- monitoring Apr 10, 2025, 05:26 AM UTC
A fix has been rolled out and we're monitoring the results.
- resolved Apr 10, 2025, 05:56 AM UTC
The incident has been resolved.
Major April 8, 2025 Difficulties for users to log in 1h 21m Platform UI
Started Apr 08, 2025, 10:17 AM UTC· resolved Apr 08, 2025, 11:39 AM UTC
- identified Apr 08, 2025, 10:17 AM UTC
We're currently investigating an issue that may be preventing some users from logging in. Our team is actively working to resolve this as quickly as possible. We appreciate your patience and will provide updates as soon as we have more information.
- monitoring Apr 08, 2025, 10:25 AM UTC
We have released a fix and are seeing successful logins.
- resolved Apr 08, 2025, 11:39 AM UTC
This incident has been resolved.
Notice February 10, 2025 Reenabled ignored run warnings –
Started Feb 10, 2025, 11:00 AM UTC· resolved Feb 10, 2025, 11:00 AM UTC
- resolved Feb 14, 2025, 12:57 PM UTC
The ignored run warnings feature has been reenabled following the resolution of the previous incident, "Degraded performance related to ignored run warnings."
Minor February 6, 2025 Degraded performance related to ignored run warnings 3h 58m GraphQL API, Event processing, Platform UI
Started Feb 06, 2025, 10:41 AM UTC· resolved Feb 06, 2025, 02:39 PM UTC
- monitoring Feb 06, 2025, 10:41 AM UTC
We have identified an edge-case in the ignored run warnings feature that may impact system performance. While we are investigating and attempting to fix this issue, we have temporarily disabled the feature. While the feature is disabled we will not display information about why runs were not triggered or why stacks are not in sync with the latest commit. We will continue to investigate the situation and will provide an update as soon as we have more information.
- resolved Feb 06, 2025, 02:39 PM UTC
We are continuing to monitor the system performance and have decided to continue to keep the ignored run warnings feature disabled temporarily. We are resolving this incident for now, and will aim to re-enable the feature as soon as possible.
Major November 27, 2024 Partial MFA unavailability 1h 50m
Started Nov 27, 2024, 01:25 AM UTC· resolved Nov 27, 2024, 03:15 AM UTC
- identified Nov 27, 2024, 01:25 AM UTC
We've identified a regression caused by a recent deployment that results in MFA not working for all types of keys. We've identified the issue and are working on a fix.
- resolved Nov 27, 2024, 03:15 AM UTC
This incident has been resolved.
Minor November 4, 2024 Degraded public worker availability 49m Public workers
Started Nov 04, 2024, 12:57 AM UTC· resolved Nov 04, 2024, 01:47 AM UTC
- investigating Nov 04, 2024, 12:57 AM UTC
We're noticing a degraded availability of workers on the public (shared) worker pool
- monitoring Nov 04, 2024, 01:47 AM UTC
Spacelift runs were failing while attempting to verify the Spacelift worker binary. The problem was caused by an expired GPG public key. The GPG key has now been updated and runs are succeeding again.
- resolved Nov 04, 2024, 01:47 AM UTC
This incident has been resolved.