Harness Outage History

Started Sep 15, 2026, 10:58 AM UTC· resolved Sep 15, 2026, 11:19 AM UTC

1. investigating Sep 15, 2026, 10:58 AM UTC

   We are currently investigating this issue.

2. resolved Sep 15, 2026, 11:19 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 15, 2026, 10:57 AM UTC· resolved Sep 15, 2026, 12:09 PM UTC

1. investigating Sep 15, 2026, 10:57 AM UTC

   We are currently investigating this issue.

2. identified Sep 15, 2026, 11:05 AM UTC

   The issue has been identified and a fix is being implemented.

Read the full incident report →

Started Sep 14, 2026, 07:00 PM UTC· resolved Sep 15, 2026, 03:28 AM UTC

1. investigating Sep 14, 2026, 10:48 PM UTC

   We are currently investigating this issue.

2. identified Sep 15, 2026, 12:42 AM UTC

   The issue has been identified and a fix is being implemented.

3. identified Sep 15, 2026, 12:45 AM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Sep 15, 2026, 01:46 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Sep 15, 2026, 03:28 AM UTC

   This incident has been resolved.

6. postmortem Sep 21, 2026, 06:26 PM UTC

   # **Summary** To improve the security posture, On September 14, 2026, Harness performed a scheduled rotation of the TLS certificate used to secure Delegate connections for accounts using mutual TLS \(mTLS\),The replacement certificate passed Harness's existing pre-deployment validation and smoke testing. However, the rotation also introduced a change in certificate authority, which required additional certificate-chain and gateway configuration this prevented affected Delegates from connecting to Harness. Harness has identified the contributing factors and is implementing the corrective actions described below to prevent a recurrence. # **Impact** Delegates configured to use mutual TLS in Prod environment were unable to connect to Harness for until Harness applied a temporary mitigation \(disabling mTLS\) that restored connectivity while the underlying issue was corrected. This affected the ability to run pipelines and other operations dependent on the affected Delegates. Full, permanent connectivity — with mTLS restored to its normal, fully secure configuration — was confirmed for both environments by 7:20 PM PST. # **Root Cause** ‌ As part of routine certificate maintenance, Harness rotated the certificate used by the gateway component that secures mutual TLS connections for Delegates in prod environments. The rotation also introduced a change in certificate authority, which required additional certificate-chain and gateway configuration that was not fully covered by the existing validation process. As a result, the gateway was unable to load the new certificate correctly, and Delegates relying on mutual TLS could not establish a secure connection. # **Remediation** Once the certificate rotation was confirmed as the source of the issue, Harness engineering worked continuously to correct the certificate and validate the fix with Citi before fully restoring the secure configuration. Corrective and Preventive Actions During remediation, Harness also identified and corrected a secondary issue with the replacement certificate's chain, which required an additional deployment before connectivity was fully and permanently restored. # **Next Steps** Harness is taking the following steps to prevent a recurrence of this issue: 1. Extend pre-deployment validation and smoke testing to explicitly cover certificate authority changes and full certificate-chain verification, closing the specific gap that allowed this configuration through. 2. Enhance automation for managing mTLS-related configuration changes, 3. Improve monitoring and alerting for Delegate connectivity, including account-level visibility, so similar issues are detected automatically

Read the full incident report →

Started Sep 11, 2026, 02:22 PM UTC· resolved Sep 11, 2026, 03:45 PM UTC

1. investigating Sep 11, 2026, 02:22 PM UTC

   We are currently investigating this issue.

Read the full incident report →

Started Sep 08, 2026, 09:52 PM UTC· resolved Sep 09, 2026, 04:30 AM UTC

1. investigating Sep 08, 2026, 09:52 PM UTC

   We are currently investigating this issue.

2. investigating Sep 08, 2026, 10:53 PM UTC

   We are continuing to investigate this issue.

3. investigating Sep 08, 2026, 10:53 PM UTC

   We are continuing to investigate the issue.

4. investigating Sep 08, 2026, 11:31 PM UTC

   We are continuing to investigate this issue.

5. identified Sep 08, 2026, 11:52 PM UTC

   The issue has been identified and a fix is being implemented.

6. identified Sep 09, 2026, 12:29 AM UTC

   We are continuing to work on a fix for this issue.

7. monitoring Sep 09, 2026, 12:36 AM UTC

   A fix has been implemented and we are monitoring the results.

8. monitoring Sep 09, 2026, 12:39 AM UTC

   A fix has been implemented and we are monitoring the results.

9. resolved Sep 09, 2026, 04:30 AM UTC

   This incident has been resolved.

10. postmortem Sep 09, 2026, 11:23 PM UTC

    ### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period. ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths

Read the full incident report →

Started Sep 05, 2026, 08:34 AM UTC· resolved Sep 05, 2026, 08:38 AM UTC

1. investigating Sep 05, 2026, 09:40 AM UTC

   We are currently investigating this issue.

2. resolved Sep 05, 2026, 09:40 AM UTC

   This incident has been resolved.

3. postmortem Sep 09, 2026, 06:22 PM UTC

   ## **Summary** Between 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager` and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended. Post restart the delegate would re-connect and the issue was resolved. ## **Root cause** On Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected. ## **Impact** All K8s delegates and \`Docker/ECS\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled. ## **Remediation** * Immediate: We have added additional monitoring and increased resources for handling the influx of traffic. * Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving. ## **Action Items** To prevent such issues from happening again, Harness will work on the following: 1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again. 2. Fix the identified delegate service managers database client reconnect failures 3. Fix the hotpots that can cause query latency.

Read the full incident report →

Started Sep 03, 2026, 10:00 AM UTC· resolved Sep 03, 2026, 11:40 AM UTC

1. investigating Sep 03, 2026, 10:00 AM UTC

   Pipeline execution is stuck in Prod1. We are investigating the issue.

2. monitoring Sep 03, 2026, 10:28 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 03, 2026, 11:40 AM UTC

   This incident has been resolved.

4. postmortem Sep 10, 2026, 07:21 AM UTC

   # Summary On September 3, 2026, between approximately 2:54 AM and 3:44 AM PDT, customers running pipelines on Prod1 experienced delays in pipeline execution graph rendering and execution status updates. Pipeline execution itself was not affected — pipelines continued to run and complete — but the visual graph and status information in the Harness UI lagged behind actual execution progress. Harness engineering identified the cause, added processing capacity, and restored normal graph and status updates. Extended monitoring confirmed full recovery the following day. # Root Cause A planned database failover activity in the Prod1 region temporarily increased network latency between the pipeline execution service and its database while traffic was briefly served cross-region. This reduced the rate at which the service could process pipeline orchestration events, and a processing backlog began to build. Pipeline execution graphs and status indicators in the UI depend on these orchestration events being processed in near real time. As the backlog grew, graph rendering and status updates fell increasingly behind actual execution progress, and some graphs had to be rebuilt rather than served from cache. Pipeline execution itself does not depend on this same processing path, so pipelines continued to run and complete throughout the incident. # Impact * Affected users: Customers with pipeline executions running on Prod1 during the incident window. * Symptom: Delayed pipeline execution graph rendering and delayed execution status updates in the Harness UI. * Not reported as impacted: Pipeline execution itself — pipelines continued to run and complete. * Data: No data loss, corruption, or exposure was identified. * Suggested workaround: None required from customers. The issue was resolved entirely by Harness engineering. * Duration of impact: Approximately 45–50 minutes of degraded graph and status visibility, from 2:54 AM to 3:44 AM PDT on September 3, 2026. Harness continued monitoring after the backlog cleared and confirmed full recovery. # Remediation ## Immediate Harness engineering added processing capacity to the affected service and restarted it, which cleared the event-processing backlog and restored normal pipeline execution graph and status updates. # Action Items | **Action Item** | **Objective** | | --- | --- | | Queue based alarms | We are raising priority for queue based alarms so we can take faster remediation actions \(increase capacity\) |

Read the full incident report →

Started Aug 28, 2026, 07:04 AM UTC· resolved Aug 28, 2026, 07:35 AM UTC

1. investigating Aug 28, 2026, 07:04 AM UTC

   We are currently investigating this issue.

2. identified Aug 28, 2026, 07:16 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 28, 2026, 07:29 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 28, 2026, 07:35 AM UTC

   This incident has been resolved.

5. postmortem Sep 02, 2026, 10:15 PM UTC

   ## Summary Between August 27 and August 28, 2026, customers experienced an issue where some pipelines, deployments, and related resources appeared as not found in the Harness UI and API, even though the underlying data remained intact. ‌ The issue occurred during a planned internal infrastructure update that affected communication between internal platform services. As a result, requests that depended on account, organization and project scope resolution were unable to complete successfully, which led to incorrect not found responses being returned to customers for existing entities. ‌ Engineering identified the issue, rolled back the change, and restored normal service. No customer data was lost or deleted during the incident. ‌ ## Root Cause The issue was caused by a configuration error introduced during a planned internal service routing update in Production. An internal platform service responsible for resolving account, organization, and project context was unable to validate requests from other Harness services after the change was applied. Because that validation step is required before many entity reads and pipeline-related actions can proceed, the failed requests surfaced to customers as not found errors for resources that continued to exist normally. The issue was limited to the affected production environment and was resolved by reverting the change and restoring the previous service communication path. ‌ ## Impact * Some customers saw existing pipelines, deployments, and related entities appear as not found in the UI and API. * Some pipeline-related operations, including execution progression, webhook-triggered starts, scheduled trigger evaluation, and entity listing, were temporarily disrupted. * The issue affected availability and visibility of existing entities, but it did not remove data or change customer configurations. * No unauthorized access occurred, and no customer data loss was observed. ## Remediation * **Immediate:** Reverted the infrastructure configuration update and restored the previously working service communication path. * **Recovery validation:** Verified that affected entity lookups, pipeline operations, and dependent APIs were functioning normally after rollback. * **Permanent:** Corrected the configuration handling associated with the update so similar issues do not interfere with service-to-service authentication in future rollouts. ## Action Items To prevent such issues from happening again, Harness will 1. Improve configuration validation by enhancing the pre-deployment tests to verify internal service communication before shifting production traffic. 2. Enhance monitoring and alerting for internal authentication failures so issues can be detected earlier. 3. Improve error handling so dependency failures are less likely to appear to customers as resource not found errors.

Read the full incident report →

Started Aug 26, 2026, 08:38 AM UTC· resolved Aug 26, 2026, 09:10 AM UTC

1. investigating Aug 26, 2026, 08:38 AM UTC

   We are currently investigating an issue reported in IACM pipelines in Prod-1 , Prod-2 ,Prod-4 and EU1 harness clusters.

2. investigating Aug 26, 2026, 08:43 AM UTC

   We are continuing to investigate this issue.

3. monitoring Aug 26, 2026, 09:05 AM UTC

   We have rollout out a fix to resolve this issue in all clusters.

4. resolved Aug 26, 2026, 09:10 AM UTC

   This incident has been resolved.

5. postmortem Sep 21, 2026, 06:07 PM UTC

   # Root Cause Analysis Report # 1. Summary On Wednesday, August 26, 2026, some customers running Infrastructure as Code Management \(IaCM\) pipelines in Harness production saw those pipelines start, sit in a Running state with no init or plan steps, then abort or expire after about 10 minutes. OpenTofu / Terraform work did not start. The failure presented as a pipeline timeout; it was not an OpenTofu timeout and not a failure of the customer's infrastructure-as-code. Other product areas that do not go through this IaCM-to-Pipeline handoff were not affected by this failure mode. # 2. Impact * Some IaCM pipelines did not complete. Stages appeared stuck, then timed out or aborted. * Execution graphs showed the IaCM stage Running with no child steps. Init and plan never ran, so there was no plan output and no apply. * A reported affected execution ran for 10 minutes and 7 seconds before being aborted. That interval is a platform stuck-execution safeguard, not a customer-configured step-timeout setting. * Customers needed to re-run their pipeline after service was restored in their environment. # 3. Root Cause IaCM stages are driven by the Harness Pipeline service. After an IaCM stage starts, IaCM must tell Pipeline to create the child steps \(init, plan, and so on\). Those messages travel on Harness's internal event bus. An incompatible modification within a shared library \(unrelated to IaCM specifically\) was inherited by the IaCM Manager upon rebuilding with the updated version, without any direct changes to IaCM functionality. # 5. Remediation IaCM Manager was rolled back to an older version in each affected production environment, All affected environments were restored by 09:00:40 UTC on August 26. Customers do not need to change pipeline YAML, OpenTofu/Terraform configuration, or connectors. Re-running the same pipeline after service was restored is sufficient. # 6. Preventive Actions 1. **Backward-compatible or atomic SDK/event changes:** producer and consumer changes in the shared Pipeline SDK and Pipeline service must remain backward compatible, or be shipped and deployed together atomically, so that mixed versions can never silently drop messages. 2. **Detection:** add alerting for an IaCM stage that stays Running with zero children, or for child-step events that are produced but never consumed, rather than relying on the time based abort as the first signal.

Read the full incident report →

Started Aug 24, 2026, 12:57 AM UTC· resolved Aug 24, 2026, 01:12 AM UTC

1. investigating Aug 24, 2026, 12:57 AM UTC

   We are currently investigating this issue.

2. investigating Aug 24, 2026, 01:01 AM UTC

   We are currently investigating reports that the Feature Management & Experimentation (FME) user interface is failing to load. Customers attempting to access the FME console may encounter errors or unresponsive pages. Feature flag evaluation and SDK traffic are not believed to be affected. A further update will follow shortly.

3. monitoring Aug 24, 2026, 01:08 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 24, 2026, 01:12 AM UTC

   This incident has been resolved.

5. postmortem Aug 27, 2026, 11:36 PM UTC

   ## Summary * Starting at **23:42 UTC** on August 23, 2026, several FME customers reported failures loading the FME UI. * FME UI Artifacts served from the CDN expired due to a retention policy, causing FME UI to fail to load. * Any flag request changes through the API, change delivery, and the data pipeline continued to work with no interruption. ## Root Cause * The FME UI is served from a CDN. The UI artifacts got evicted due to a retention policy, causing the UI to fail to load for all users. ## Impact * The FME UI was unable to load for all users across all production environments. ### What was not impacted? * SDK functionality and runtime flag evaluation * Admin API calls * Customer flag configuration data * No data loss occurred ## Remediation * FME UI got restored in the CDN through a deployment * Recovery confirmed across all production environments before closing the incident. ## Action Items * Improve the asset retention policy so that the currently active version is never subject to eviction.

Read the full incident report →

Started Aug 20, 2026, 03:37 PM UTC· resolved Aug 20, 2026, 07:23 PM UTC

1. investigating Aug 20, 2026, 03:37 PM UTC

   We are currently investigating this issue.

2. investigating Aug 20, 2026, 04:05 PM UTC

   The slowness could cause either of below symptoms: - Pipelines not starting - Delays in execution - Pipelines being cancelled due to timeouts

3. investigating Aug 20, 2026, 04:22 PM UTC

   Our cloud provider is facing an active incident and we are following up.

4. investigating Aug 20, 2026, 04:32 PM UTC

   We are continuing to investigate this issue.

5. identified Aug 20, 2026, 04:50 PM UTC

   Our cloud provider has confirmed an ongoing incident impacting multiple regions. Harness pipelines have not experienced failures as a result, though some users may continue to experience slowness. We are monitoring the situation closely and will provide updates as more information becomes available.

6. monitoring Aug 20, 2026, 05:17 PM UTC

   We are observing better latencies across the board due to the fix at the cloud provider end. We are keeping a close watch.

7. resolved Aug 20, 2026, 07:23 PM UTC

   This incident has been resolved.

8. postmortem Aug 26, 2026, 04:57 AM UTC

   # Summary On 20 August 2026, beginning at approximately 15:00 UTC, the Harness platform experienced widespread performance degradation across all production environments. Pipeline executions that normally complete in around two minutes took seven to ten minutes. Continuous Delivery, Continuous Integration, pipeline orchestration, and Feature Management & Experimentation were all affected. Google Cloud Platform experienced a multi-product incident in the us-west1 region affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk I/O. Harness production infrastructure runs on persistent disks in that region. The degradation raised database operation latency from approximately 2 ms to over 10 ms at the 95th percentile, which in turn caused message-queue processing lag and propagated to every service that depends on timely database access. ‌ # Impact This was a degradation, not an outage. Pipelines continued to execute and complete successfully throughout; they were slow rather than failing. No data was lost, and no customer work was dropped as a result of this incident. # **Root cause** Harness production infrastructure in the affected environments runs on Google Cloud Platform persistent disks in the us-west1 region. When that storage layer degraded, the effect propagated through the platform in a predictable chain: **Persistent-disk I/O degradation in us-west1.** Google Cloud Platform experienced a multi-product incident affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk performance. This was an infrastructure failure in the provider’s environment, outside Harness’s control. # **Preventive actions** Although Harness cannot prevent a cloud provider infrastructure failure. The actions below are aimed at detecting one faster and being better positioned to act on it. | **Action** | | --- | | Continue routine pre-testing of targeted cross-region database failovers, as performed during this incident, to keep failover readiness verified rather than assumed | | Assess full-stack multi-region failover readiness for future scenarios in which cross-region latency would be unacceptable |

Read the full incident report →

Started Aug 20, 2026, 02:32 PM UTC· resolved Aug 20, 2026, 03:02 PM UTC

1. investigating Aug 20, 2026, 02:32 PM UTC

   We are currently investigating this issue.

2. identified Aug 20, 2026, 02:47 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 20, 2026, 02:56 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 20, 2026, 03:02 PM UTC

   This incident has been resolved.

5. postmortem Aug 23, 2026, 09:45 PM UTC

   ### Summary On August 20, 2026, between 10:24 and 14:55 UTC, a subset of FME writes failed. Writes made from the FME UI and writes made with Harness access tokens \(PATs and SATs\) were not affected. Runtime flag evaluation continued to work normally. The issue was mitigated by reverting a recent authentication change in a shared governance service, and affected writes returned to normal by 14:55 UTC. Status: [https://status.harness.io/incidents/rhthgm7d5dkz](https://status.harness.io/incidents/rhthgm7d5dkz) ### Root Cause A change in how a shared governance service authenticated inbound calls resulted in some FME writes being rejected. Those writes used service-to-service credentials that the governance service could no longer verify after the change. FME surfaces a governance failure to the client as HTTP 499, the same status used when a governance policy intentionally denies a change. Because 499 is a valid, expected response in that deny path, the failures did not look like an outage on our alerts, and the incident was identified from customer reports rather than internal detection. ### Impact * A subset of FME writes failed during the window, primarily those made using legacy Split API keys or change request scheduling. * Writes made from the FME UI were not impacted. * Writes using Harness access tokens \(PATs and SATs\) were not impacted. * Runtime flag evaluation continued normally. * No data loss occurred. Failed writes did not apply. ‌ ### Remediation Reverted the governance-service authentication change. Affected writes returned to normal immediately. ### Action Items To prevent such issues from happening again, * Harness will return a distinct error \(not 499\) when a write fails because governance could not be evaluated, so it is not confused with an intentional policy denial. * Add alerting on the governance evaluation call itself, rather than relying on the client-facing status code. * Expand authentication support for policy evaluations. * Expand automated coverage for additional write scenarios.

Read the full incident report →

Started Aug 19, 2026, 01:22 PM UTC· resolved Aug 19, 2026, 04:07 PM UTC

1. investigating Aug 19, 2026, 01:22 PM UTC

   We are currently investigating this issue.

2. investigating Aug 19, 2026, 02:03 PM UTC

   We are continuing to investigate this issue.

3. investigating Aug 19, 2026, 02:55 PM UTC

   We are continuing to investigate this issue.

4. identified Aug 19, 2026, 03:13 PM UTC

   The issue has been identified and a fix is being implemented.

5. monitoring Aug 19, 2026, 03:22 PM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Aug 19, 2026, 03:33 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Aug 19, 2026, 04:07 PM UTC

   This incident has been resolved.

8. postmortem Aug 26, 2026, 05:20 PM UTC

   **Summary** On 19 August 2026 between 12:35 and 17:29 UTC, the Harness Application Security service experienced a significant disruption affecting both the customer-facing console and the data ingestion pipeline in the SaaS Production and US1 regions. ‌ **Root Cause** The internal configuration service that supplies runtime settings to nearly every other component became overloaded and entered a repeated restart cycle. Because so many services depend on it, the effects were broad: console pages such as protection policies, posture views, activity logs, API inventory, and custom policy failed to load or timed out, and downstream processing stalled while waiting for configuration it could not obtain. # **Customer impact** | **Dimension** | **Detail** | | --- | --- | | Console \(UI\) impact | Multiple pages failed to load or timed out, including protection policies, posture event pages and posture views inside dashboards and insight pages, activity log queries, API inventory screens, custom policy, and sensitive-data views and widgets. | | Ingestion impact | Security telemetry processing degraded severely and, in some paths, stopped entirely. Consumer lag grew across normalisation, grouping, anomaly detection, generation, and related processing stages. | | Data loss | A subset of telemetry ingested during the disruption was permanently dropped. | ‌ **Mitigation** Several intermediate mitigations additional CPU and memory, relaxed health-check thresholds, a database restart, and a larger connection pool ameliorated the issue. Disabling the new feature in both affected regions restored throughput sharply and durably. The incident was resolved at 17:29 UTC. ‌ # **Preventive actions** The following actions are committed and tracked internally to completion. The feature that triggered this incident remains disabled and will not be re-enabled until the work below is complete and validated. | **Action** | | --- | | | | OPtimize the code by tuning parameters such as cache eviction and retention , evaluate cursor-based pagination for bulk rule retrieval as rule counts grow | | Add a purpose-built database index for the service-scoping access pattern | | Remediate pipeline recovery semantics so consumers replay safely after position-marker loss instead of skipping backlog | | Mandate staged rollout for configuration overrides that alter downstream request patterns: low-volume cluster, then mid-volume, then high-volume | | Add backpressure and concurrency protection to the configuration service: circuit breaking, bounded queues, and timeout isolation | | Enhance observability by Instrumenting more detailed metrics |

Read the full incident report →

Started Aug 06, 2026, 02:30 PM UTC· resolved Aug 06, 2026, 04:01 PM UTC

1. investigating Aug 06, 2026, 02:30 PM UTC

   We are currently investigating a Harness component that is experiencing issues. We are working to identify the cause and restore normal operations as soon as possible.

2. investigating Aug 06, 2026, 02:32 PM UTC

   We are continuing to investigate this issue.

3. identified Aug 06, 2026, 03:46 PM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Aug 06, 2026, 04:01 PM UTC

   This incident has been resolved.

5. postmortem Aug 10, 2026, 06:24 PM UTC

   ## Summary Customers on Prod1, Prod2, and Prod3 \(US\) clusters experienced failures when loading SEI 2.0 dashboards on August 6, 2026, from 7:22 AM PDT to 9:03 AM PDT. Customers calling the SEI 2.0 API also experienced similar failures. No customer data was lost, and ingestion of all integration data continued to work uninterrupted. SEI customers using 1.0 were not impacted. ## Root Cause The incident was caused by resource exhaustion on the nodes serving queries. This resource degradation developed in a pattern that did not cross our existing alerting thresholds early enough to provide sufficient warning or allow mitigation before customer impact occurred. ## Impact Customers on Prod1, Prod2, and Prod3 \(US\) clusters were unable to load SEI 2.0 dashboards during the incident window. **Duration:** August 6, 2026, 7:22 AM PDT – 9:03 AM PDT \(~1 hour 41 minutes\) ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the root cause, our team took immediate corrective action by adding capacity to restore the affected systems. Services were fully recovered, and all dashboards resumed normal operation at 9:03 AM PDT. ## Action Items To prevent from such issues happening again, Harness is/has Proactively added capacity updates have been applied to prevent this issue from recurring #### 2. Enhanced Monitoring and Alerting Additional monitoring and alerting have been put in place to detect anomalies early, focused on a leading indicator, which in this case was thread pool exhaustion, before they can impact dashboard availability and data rendering. #### 3. System Patch in Progress We are working with our vendor to apply a patch to remediate this and similar issues completely.

Read the full incident report →

Started Aug 06, 2026, 01:50 PM UTC· resolved Aug 06, 2026, 06:21 PM UTC

1. monitoring Aug 06, 2026, 01:50 PM UTC

   We are monitoring the stuck pipelines in prod2. The new executions are passing as we are continuously monitoring the services.

2. monitoring Aug 06, 2026, 04:58 PM UTC

   We are monitoring the stuck pipelines in prod2. For the customers who are still seeing stuck pipelines, we request you to abort and re-trigger.

3. resolved Aug 06, 2026, 06:21 PM UTC

   This incident has been resolved.

4. postmortem Aug 17, 2026, 06:40 PM UTC

   ## **Summary** On August 6, 2026 \(morning PDT\), some customers running pipelines in the Prod2 production environment observed pipeline executions that stopped making progress — stages that did not advance and produced no further output or status updates. The issue was reported by affected customers. Harness engineers identified the cause, mitigated the impact, and pipeline executions returned to normal operation. The issue was caused by a self-referential pipeline expression. A Git webhook triggered a pipeline that referenced the contents of the webhook payload, and the payload itself contained further copies of that same expression. Each round of expression resolution therefore produced more expressions to resolve, doubling the amount of work each time. This exhausted the resources of the service instance processing that execution, and other executions assigned to the same instance were unable to progress while it was in that state. ## **Impact** During the incident window \(approximately 6:11 AM to 11:23 AM PDT on August 6, 2026\): * Some customers' pipeline executions on Prod2 stalled mid-execution and made no further progress. * Affected executions produced no new step output or status updates, and had to be aborted and re-run after mitigation. * Behavior was limited to executions being processed by the affected service instance — pipelines handled by other instances continued to execute normally. There was **no data loss**. Pipeline definitions, execution history, and stored state were unaffected. The majority of pipelines on Prod2 continued to execute successfully throughout the incident; the primary impact was that some in-flight executions could not complete and needed to be re-run once the issue was mitigated. ## **Root Cause** Harness pipelines support expressions that are resolved at runtime — for example, an expression that inserts the contents of the Git webhook payload that triggered the pipeline. In this case, a Git commit message contained the literal text of the payload expression itself, twice, and the pipeline referenced that same payload expression. Because the commit message is part of the webhook payload, resolving the expression inserted the entire payload — including the two literal copies of the expression carried in the commit message. Those newly inserted copies were then treated as expressions to be resolved, and each pass inserted two more full copies of the payload. The size of the value being processed, and the work required to process it, therefore doubled on every pass and grew exponentially rather than converging. Harness has a safeguard intended to stop exactly this: expression resolution is bounded by a maximum nesting depth, beyond which resolution halts and the pipeline fails with an explicit error. A defect in that safeguard meant the limit was not applied in this specific self-referential case, so resolution continued unchecked. Expression resolution runs inline on the threads that start pipeline steps. As each pass consumed progressively more memory and CPU without ever completing, the service instance performing that work stopped making progress, and every execution assigned to that instance stalled — which is what customers reported. ## **Mitigation** Harness completed the following immediate mitigation steps: * Identified the pipeline and the expression pattern responsible for the runaway resolution. * Stopped the affected service instance so that it would take on no further work. The remaining healthy instances picked up and processed queued executions normally. * Confirmed that pipeline executions returned to normal and closed the incident. These actions restored normal pipeline execution behavior and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented: * Fix the defect in the expression depth and loop-detection safeguard so that self-referential expressions are caught and fail fast with a clear error instead of consuming resources without bound. * Prevent payload expressions from being resolved out of trigger payload content, removing the self-referential path entirely. * Tighten the maximum expression nesting depth and evaluate explicit loop detection in addition to the existing depth limit. * Enhance automated tests in pre-production environments that reproduce self-referential expression patterns and verify that the safeguard detects and stops them. * Add monitoring for this pattern in pipeline executions so that it is detected proactively.

Read the full incident report →

Started Aug 04, 2026, 12:07 PM UTC· resolved Aug 04, 2026, 03:28 PM UTC

1. investigating Aug 04, 2026, 12:07 PM UTC

   We are currently investigating this issue.

2. investigating Aug 04, 2026, 12:08 PM UTC

   We are continuing to investigate this issue.

3. investigating Aug 04, 2026, 12:17 PM UTC

   We are continuing to investigate this issue.

4. investigating Aug 04, 2026, 01:00 PM UTC

   We have identified the issue and started to implement the fix , prod2 is restored.

5. investigating Aug 04, 2026, 01:09 PM UTC

   We are continuously rolling out fixes to all Clusters, Prod EU1 has been resolved.

6. investigating Aug 04, 2026, 01:16 PM UTC

   We are continuously rolling out fixes to all Clusters, Prod3 has been resolved.

7. resolved Aug 04, 2026, 03:28 PM UTC

   This incident has been resolved.

8. postmortem Aug 06, 2026, 05:02 PM UTC

   # Executive Summary On August 4, 2026, between approximately 3:36 PM and 9:00 PM IST, customers using Infrastructure as Code Management \(IaCM\) on Prod0 and Prod1 were unable to access the Variable Sets settings page. The page rendered blank with no error message, and customers with Variable Sets attached to their workspaces could not view or manage them for the duration of the incident. Prod2, Prod3, and EU1 were not affected. Separately, during the same window, a scheduled maintenance action caused the IaCM settings tab to temporarily disappear across all environments. This was identified and reversed within the incident bridge call before significant customer impact occurred. We deployed a hotfix that restored full access to the Variable Sets page on Prod0 and Prod1 the same evening, and we are implementing permanent safeguards described below to prevent this class of issue from recurring. # Impact * Customers with the Variable Sets feature enabled on Prod0 and Prod1 were unable to view or manage Variable Sets for approximately 5–6 hours. * No data was lost or corrupted, this was a UI routing failure only; underlying Variable Sets data and configuration were not affected. * Prod2, Prod3, and EU1 were not affected by this issue. * A secondary issue, a scheduled feature flag operation caused the IaCM settings tab to temporarily disappear across all environments during the incident bridge call. This was identified and reversed within minutes. External customer exposure for this secondary issue is still being confirmed. # Root Cause A platform routing change released on July 18, 2026 updated how IaCM settings pages are resolved in the user interface. As part of that change, any settings page that had not been explicitly re-registered in the new routing structure became unreachable. The Variable Sets page had not been re-registered under the new routing structure, making it inaccessible in the environments where the routing change had been deployed — Prod0 and Prod1. Because the failure occurred at the routing layer rather than within the page itself, the page rendered blank with no visible error rather than showing a clear failure message. # Remediation ## Immediate We deployed a hotfix that re-registered the Variable Sets page in the updated routing structure, restoring access for all affected customers on Prod0 and Prod1. ## Permanent We are adding automated end-to-end tests that navigate to settings pages with relevant feature flags enabled, configured as a required gate in our release pipeline. We are also documenting and enforcing the routing constraint through static analysis so that settings pages are never inadvertently left out of the routing structure during future platform changes. # Action Items To prevent such issues from happening again, 1. Enhance automated tests, that navigate to settings pages with relevant feature flags enabled, configured as a blocking gate in the release pipeline, so this class of regression is caught before it reaches production. 2. Establish an explicit checklist step for future platform-wide architectural changes that verifies all existing settings pages remain accessible in the updated routing structure before the change is promoted to production.

Read the full incident report →

Started Aug 03, 2026, 03:08 AM UTC· resolved Aug 03, 2026, 05:53 AM UTC

1. investigating Aug 03, 2026, 03:08 AM UTC

   We are currently investigating this issue.

2. monitoring Aug 03, 2026, 03:37 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 03, 2026, 05:53 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 31, 2026, 08:22 PM UTC· resolved Aug 01, 2026, 09:11 AM UTC

1. investigating Jul 31, 2026, 08:22 PM UTC

   We are currently investigating this issue.

2. identified Aug 01, 2026, 02:41 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 01, 2026, 05:45 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 01, 2026, 09:11 AM UTC

   This incident has been resolved.

5. postmortem Aug 21, 2026, 04:19 AM UTC

   # **Summary** Between 25 July and 4 August 2026, pipeline execution dashboards and overview pages in the Harness Prod 2 and Prod 3 clusters displayed data that was between behind real time. Pipelines themselves continued to build, deploy, and execute normally throughout; the issue was confined to how quickly execution records were copied into the database that serves reporting and dashboard views. ‌ **No customer data was lost.** Every affected record remained durably stored and was replayed into the analytics datastore once the underlying limitation was removed. Harness migrated the affected clusters to a horizontally scalable, queue-backed version of the replication component on 1 August 2026 and completed targeted data backfills for all affected accounts. # **Root cause** Harness maintains a change-data-capture component that continuously replicates pipeline execution records from the primary operational datastore into a separate time-series datastore optimised for dashboards and reporting queries. Dashboards read exclusively from the analytics datastore. When replication falls behind, dashboards render an accurate but older view of the world, while execution itself is unaffected. This was caused by sharp, sustained increase in database write volume from another Harness platform module sharing the same replication path exceeded the throughput ceiling of the older, single-instance version of that component still running in Prod 2 and Prod 3. A backlog formed and grew. ‌ ‌ # **Preventive actions** Harness has completed or committed to the following actions to prevent such issues. | **Action** | | --- | | Fine tune the replication lag alerting so that any delay beyond a defined threshold is notified | | Add a replication lag panel to the standard platform monitoring board so pipeline health is visible to on-call by default | | Reduce write amplification from co-tenant modules through per-module rate limiting or entity filtering on the replication stream |

Read the full incident report →

Started Jul 31, 2026, 01:48 PM UTC· resolved Aug 01, 2026, 11:49 PM UTC

1. investigating Jul 31, 2026, 01:48 PM UTC

   We are currently investigating this issue.

2. identified Aug 01, 2026, 02:34 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 01, 2026, 02:44 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 01, 2026, 11:49 PM UTC

   This incident has been resolved.

5. postmortem Aug 13, 2026, 12:50 AM UTC

   # **Summary** On July 31, 2026, artifact uploads performed through pipeline in the EU1 cluster began failing with an authentication error. Uploads initiated manually \(outside of a pipeline\) were not affected, and the ability to retrieve existing artifacts \(downloads\) was also unaffected — this was isolated to the specific pipeline upload path in one cluster. # **Impact** * Artifact uploads performed through pipeline in the EU1 cluster failed with an authentication error for approximately 4 hours and 34 minutes. * Retrieving existing artifacts \(downloads\) was not affected. * Manually uploading artifacts outside of a pipeline was not affected. * Other clusters/regions were not affected by this issue. # **Root Cause** The component responsible for handling pipeline-based artifact uploads is distributed as a container image. In the EU1 cluster, this image is retrieved from an internal registry that mirrors a public image source; in other clusters, the same image is retrieved directly from the public source. A publishing error in our release process caused a new build of this component to be published using a version label that was already in use, rather than being assigned a new, unique version. As a result, two different images ended up associated with the same version label in the public source. Our internal registry mirrors images from the public source via an automated replication process. Because of how that replication was triggered, it copied the original \(earlier\) image associated with that version label rather than the corrected one. This meant the EU1 cluster — which pulls from the internal mirror — ended up running a different, defective image than other clusters, which pull directly from the public source and therefore received the corrected image. The defective image contained an authentication issue that caused pipeline uploads to fail. # **Mitigation** * Reverted the affected account to the last known-good version of the upload component, immediately restoring pipeline uploads. * Published a corrected, permanent version of the component to resolve the issue across all clusters. # **Next steps** ‌ * Fix the upload step to remove the underlying container-related defect that made this failure mode possible. * Update our release pipeline for this component so that publishing an image can never overwrite an existing version — every publish must create a new, distinct version going forward.

Read the full incident report →

Started Jul 30, 2026, 05:59 AM UTC· resolved Jul 31, 2026, 03:22 AM UTC

1. investigating Jul 30, 2026, 05:59 AM UTC

   Summary - We are intermittently facing network connectivity issues with our Build VM's unable to connect to external resources. We are currently investigating the issue.

2. monitoring Jul 30, 2026, 08:07 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Jul 30, 2026, 08:07 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Jul 31, 2026, 03:22 AM UTC

   This incident has been resolved.

5. postmortem Aug 07, 2026, 07:32 PM UTC

   ## Summary Starting on August 4, 2026, CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways. ## Impact * CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services \(e.g., GitHub, Bitbucket\) over our outbound network gateways. * The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume. * No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue. * us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue. ## Root Cause ‌ Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details \(source/destination address and port\). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways ‌ ## Action Items To prevent such issues from happening again Harness will, Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..

Read the full incident report →

Started Jul 27, 2026, 09:09 AM UTC· resolved Jul 27, 2026, 01:33 PM UTC

1. investigating Jul 27, 2026, 09:09 AM UTC

   We are currently investigating this issue.

2. monitoring Jul 27, 2026, 09:22 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 27, 2026, 01:33 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 27, 2026, 06:42 AM UTC· resolved Jul 27, 2026, 01:33 PM UTC

1. investigating Jul 27, 2026, 06:42 AM UTC

   We are currently investigating this issue.

2. identified Jul 27, 2026, 06:45 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 27, 2026, 06:49 AM UTC

   A fix has been implemented and we are monitoring the results.

4. investigating Jul 27, 2026, 07:52 AM UTC

   We are currently investigating this issue.

5. investigating Jul 27, 2026, 08:13 AM UTC

   We are continuing to investigate this issue.

6. identified Jul 27, 2026, 09:04 AM UTC

   The issue has been identified and a fix is being implemented.

7. identified Jul 27, 2026, 09:05 AM UTC

   We are continuing to work on a fix for this issue.

8. monitoring Jul 27, 2026, 09:07 AM UTC

   A fix has been implemented and we are monitoring the results.

9. monitoring Jul 27, 2026, 09:07 AM UTC

   We are continuing to monitor for any further issues.

10. monitoring Jul 27, 2026, 09:09 AM UTC

    We are continuing to monitor for any further issues.

11. resolved Jul 27, 2026, 01:33 PM UTC

    This incident has been resolved.

12. postmortem Aug 07, 2026, 06:15 PM UTC

    # Summary During a recent production deployment, a defect in our internal deployment tooling caused two critical services to run with incorrect, non-production configuration values in our production environment This led to a related set of four distinct symptoms: incorrect configuration behavior, intermittent login/access failures, a filestore access issue affecting one customer environment, and delayed pipeline status updates in the UI. ‌ We have identified and are implementing a permanent fix for the underlying configuration defect, and have already put in place resource and capacity changes that resolve the UI delay symptom. ‌ At no point during this incident were pipeline executions themselves lost, corrupted, or left in a stuck state. Where execution behavior was affected, it was limited to delays in status visibility, not in the underlying processing. # Incident Details ## Incorrect Production Configuration Values Applied Our engineering team confirmed a defect in the Service Manager deployment pipeline that caused certain production services to be deployed using configuration values intended for a different environment, rather than the correct production configuration. ‌ **Root Cause** ‌ The service responsible for fetching configuration overrides during deployment queries an internal API that returns a maximum of 1,000 results per request. The total number of services in the environment recently grew beyond that limit. As a result, any service beyond the first 1,000 returned was not included in the response, and the deployment pipeline silently fell back to default configuration values for those services. This is a confirmed pagination defect in the deployment tooling, not an issue with the configuration values themselves. ‌ **Resolution** ‌ Engineering has confirmed the mechanism and is implementing a permanent fix to remove this limit-related gap in the deployment pipeline. ## Intermittent Login / Access Failures During the Service Manager deployment referenced above, some users experienced intermittent login or access failures. Under normal operation, previously running instances should continue serving traffic without interruption while a new deployment is in progress. In this incident, that fallback behavior did not occur as expected, contributing to access failures during the deployment window. ## Filestore Access Issue A filestore access issue was identified that was specific to the Prod-3 environment and affected a single customer's environment. **Root Cause** This is related to an IAM / storage-bucket permission configuration on Service Manager, potentially triggered by rollback activity. ## Delayed Pipeline Execution Status Updates in UI Some users observed that the pipeline execution graph in the UI was slow to refresh and did not reflect the latest status promptly. Importantly, this was a visibility delay only: there was no impact to actual pipeline executions, and no executions were stuck or failed as a result of this issue. **Root Cause** The pipeline execution graph relies on a message stream \(the orchestration log\) to receive status updates. During the incident window, consumer processing of this stream fell behind \(high consumer lag\), which delayed how quickly status updates reached the UI. This was caused by the fact that the underlying database was in the middle of a planned scaling operation at the same time, and a traffic spike during that window further exacerbated the delay. Users experienced this as apparent pipeline slowness, even though the underlying executions were running normally. **Resolution** We have increased resource capacity for the affected components to maintain more than 50% spare headroom going forward, reducing sensitivity to similar load spikes. This change has been implemented and is currently being validated as part of longer-term hardening for this part of the platform. # Impact Summary * Service Manager and License Manager ran with incorrect configuration values in the Prod-1 and Prod-3 environments. * Some users experienced intermittent login or access failures during the affected deployment window. * One customer environment in Prod-3 experienced a filestore access issue. * Users across affected environments saw delayed pipeline execution status updates in the UI; underlying pipeline executions continued to run correctly and were not lost, stuck, or corrupted. # Preventive Actions The following corrective and preventive actions have been identified. ‌ | **Corrective / Preventive Action** | | --- | | Correct the pagination limit in the configuration-lookup service so that all services are returned and evaluated, regardless of total count. | | Add safeguards so that a service which cannot retrieve its configuration fails safely \(e.g. alerts and blocks the deployment\) rather than silently falling back to non-production defaults. | | Increase Postgres and messaging-pipeline resource headroom \(target: greater than 50% spare capacity\) to reduce sensitivity to concurrent load and scaling events. | ‌ _We recognize the impact this incident had across multiple areas of the platform and appreciate your patience as we work through a complete resolution._

Read the full incident report →

Started Jul 22, 2026, 02:55 PM UTC· resolved Jul 22, 2026, 11:16 PM UTC

1. investigating Jul 22, 2026, 02:55 PM UTC

   We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.

2. identified Jul 22, 2026, 02:58 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 22, 2026, 03:25 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 22, 2026, 11:16 PM UTC

   This incident has been resolved.

5. postmortem Aug 10, 2026, 06:17 PM UTC

   ## Summary Customers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously the issue manifested as sporadic failures rather than a full outage. No customer data was lost. SEI 1.0 customers were not impacted. ## Root Cause Over time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident. ## Impact Customers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards. **Duration:** July 22, 2026, 07:58 PDT – 16:16 PDT \(~8 hours 18 minutes\), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward. ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes: * Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure. * Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables. * Increased capacity on the affected database nodes to provide additional headroom. These changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT. ## Action Items To prevent recurrence, we are implementing the following: 1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files. 2. We have rolled out automated compaction jobs for newly introduced tables to prevent delete file accumulation going forward.

Read the full incident report →

Started Jul 17, 2026, 05:16 PM UTC· resolved Jul 17, 2026, 09:56 PM UTC

1. investigating Jul 17, 2026, 05:16 PM UTC

   We are currently investigating this issue.

2. identified Jul 17, 2026, 05:34 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 17, 2026, 06:12 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 17, 2026, 09:56 PM UTC

   This incident has been resolved.

5. postmortem Aug 04, 2026, 02:48 AM UTC

   # Summary On 17 July 2026, following a routine code deployment, customers on older delegate versions \(858xx and below\) began experiencing delayed CI builds on Harness Cloud-hosted builds using our global build-queueing capability. Affected builds experienced an unexpected pause of up to approximately 8 minutes at the "waiting for infrastructure" stage before continuing, rather than proceeding within the expected sub-second time. Overall build slowness was intermittent. # Impact * All CI builds were potentially subject to delay; impact was most pronounced for builds on Harness Cloud-hosted infrastructure using the global build-queueing feature. * Affected builds experienced an unexplained pause of up to approximately 8 minutes before continuing, followed by a slower "cold start" since a pre-reserved compute slot was not available — this presented to users as slow builds rather than build failures. * Accounts running on newer delegate versions \(858xx and above\) were not impacted. * No builds failed outright as a direct result of this issue, and no data was lost. # Root Cause The root cause was an internal code change that inadvertently broke how a specific build-queueing record was read back from our database once builds that had already been queued under the previous version of the code encountered the newly deployed version. We resolved the immediate impact by cleaning up the affected records and reverting the underlying code change, and we are implementing several safeguards to prevent this class of issue from recurring. ‌ # Next Steps We assess the risk of a similar recurrence as low as the following actions are being understaken.The specific code path that caused this incident has already been reverted, and we are implementing structural safeguards so that this general class of issue cannot recur, regardless of where in the codebase it might otherwise occur. | **Corrective / Preventive Action** | | --- | | Add explicit, stable identifiers to all internal data classes that get stored in our database, so that future internal code reorganizations cannot break the system's ability to read back previously stored records. | | Introduce rollback and backward-compatibility testing in our pre-production environment, specifically designed to catch this class of issue before it reaches production. |

Read the full incident report →

Started Jul 17, 2026, 10:30 AM UTC· resolved Jul 17, 2026, 02:16 PM UTC

1. investigating Jul 17, 2026, 01:10 PM UTC

   We are currently investigating this issue.

2. identified Jul 17, 2026, 01:12 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 17, 2026, 01:19 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Jul 17, 2026, 01:19 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Jul 17, 2026, 02:16 PM UTC

   This incident has been resolved.

6. postmortem Jul 22, 2026, 05:25 PM UTC

   # Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.

Read the full incident report →