Harness Outage History

## **Summary** On April 27, 2026, customers running pipelines in the Prod3 environment experienced intermittent slowness in pipeline execution and delays in execution status updates in the UI. It was caused by a unexpected spike causing contention on a backend database supporting pipeline orchestration. The issue was mitigated and fully resolved. ## **Impact** **Incident window:** April 27, 2026, 1:00 PM – 3:12 PM PDT * Pipeline executions ran slower than normal; some executions took longer than expected to complete. For pipelines with stricter timeouts, there could be failures. * No widespread pipeline failures were observed * Execution view in the UI lagged behind real-time pipeline progress There was no data loss. The majority of pipelines continued to execute successfully, with the primary impact being increased latency and delayed UI updates. ## **Root Cause** Pipeline orchestration relies on a backend database to track execution state and power the execution view in the UI. During the incident, we had a spike of load, leading to increased query latency across the orchestration layer.This resulted in a backlog, causing UI updates to lag behind actual pipeline execution until the system was scaled. ## **Remediation** **Immediate Mitigation** * Scaled up the affected database instance to increase CPU capacity * Reduced query latency and eliminated lock contention * Cleared the execution-view update backlog within ~30 minutes These actions restored normal pipeline performance and UI responsiveness. ## **Action Items** To prevent such issues from happening again. * **Capacity Improvements:**Updated Prod3 capacity baseline to prevent similar resource constraints * **Proactive Detection:** Enhancing monitoring and alerting for backend resource utilization, lock contention, and critical query latency

### Summary On April 24, 2026, a large non-batched bulk DELETE operation on the prod-2 primary database triggered lock contention, causing Feature Flag API latency and hung queries across multiple customer SDKs. ### Impact 1. Slow SDK auth/init — SDKs took longer than expected to complete evaluations 2. Elevated latency across many FF APIs 3. Limited to Feature Flag module, prod-2 4. No Data loss ### Root Cause A background cleanup job executed a non-batched, single-transaction delete causing lock contention and API latency spikes **Mitigation** Immediately terminated the offending queries. ### Next Steps / Action Items To prevent such issues from happening again. we are working on 1. Enhanced alerting and observability on long running queries. 2. Permanently replace large single-transaction delete pattern with smaller batched deletes

## **Summary** On April 19, 2026, Terraform-based IaCM pipelines failed across production environments due to an issue with Terraform binary verification during runtime. The issue was caused by an expired OpenPGP signing key in a third-party library used to validate Terraform downloads. This resulted in failures when pipelines attempted to install Terraform dynamically. ## **Impact** * Terraform-based IaCM pipelines failed during execution * Failures occurred at runtime when attempting to download/verify Terraform binaries * **Customers Unaffected:** * OpenTofu-based pipelines * Pipelines using pre-installed or cached Terraform binaries Customers pinning plugin versions older than the fixed release continued to experience failures until upgraded. ## **Root Cause** The IaCM Terraform plugin relies on a third-party library \(HashiCorp’s `hc-install`\) to download and verify Terraform binaries. * The library contained a **hardcoded OpenPGP signing key** * This key **expired**, causing verification failures during Terraform installation * HashiCorp had not yet released an updated version with a renewed key ## **Remediation** ### **Immediate Mitigation** * Released **IaCM Terraform plugin v0.214.0** * Modified behavior to: * **Bypass the expired signature verification step** * Continue secure downloads over HTTPS ### **Resolution** * Rolled out the fix across **prod0–prod4** * Pipeline execution functionality was restored ## **Customer Actions Required** * Customers using pinned plugin versions **older than v0.214.0** must: * **Upgrade to v0.214.0 or later** * No action required for customers using default/latest plugin versions ## **Prevention & Next Steps** We are implementing the following improvements: * **Dependency Monitoring** * Proactive monitoring for third-party certificate/key expirations * **Upstream Coordination** * Track HashiCorp release for updated signing key * Re-enable signature verification once available * **Customer Communication** * Notify customers using older pinned versions * **Operational Improvements** * Enhance validation of external dependencies in runtime workflows

On April 16, 2026, Hosted CI Linux pipelines experienced intermittent initialization failures due to an upstream outage affecting package repositories used during environment setup. ### **Impact** A subset of customers running Hosted CI pipelines encountered failures during the initialization phase, preventing jobs from starting successfully. * Affected: 24 accounts * Failed executions: 129 pipelines * Impact duration: ~5 hours 49 minutes ### **Root Cause** The issue was caused by a service disruption in an external package repository provider. During CI environment provisioning, dependency installation requests to this upstream service timed out, causing initialization failures. ### **Remediation** **Immediate Mitigation** We updated runner configurations to bypass dependency installation during initialization and rolled out updated environments across affected clusters. **Permanent Fix** We have improved resilience in our CI infrastructure by: * Using pre-configured environments with required dependencies pre-installed * Eliminating runtime dependency on external package repositories during initialization * Enhancing failure handling for external dependency timeouts ### **Action Items / Next Steps** * Continue improving isolation from external dependencies during environment startup * Strengthen monitoring and alerting for upstream service degradation * Optimize rollout speed for infrastructure changes to reduce mitigation time

## **Summary** On April 15, 2026, between approximately 23:21 UTC and 01:58 UTC, customers using Feature Flag in the prod2 environment experienced delays in feature flag updates. Feature flag changes made via UI or API were successfully processed but were **not immediately reflected**, causing stale flag values to be served. ## **Impact** * **Scope:** Customers on **prod2 environment only** * **Customer Impact:** * Feature flag updates were **delayed or appeared ineffective** * Applications continued serving **stale configurations** * **Other Environments:** No impact to prod0, prod1, or other regions ## **Root Cause** The issue was caused by replication lag in the read replica database used for serving feature flag reads. A **long-running read query** on the replica blocked replication updates from the primary database. This caused a delay in propagating recent feature flag changes to read queries ### **What triggered the issue** * A high-volume API usage pattern involving **large paginated queries on target data.** These queries became **resource-**intensive impacting the database. ## **Mitigation** ### **Immediate Actions** * Identified and **terminated long-running queries** on the replica * Replication resumed and flag updates began reflecting correctly ## **Prevention & Next Steps** We are continuing to strengthen reliability through: * We configured replica to **automatically cancel queries** that block replication beyond a threshold and tuned **query timeouts** for heavy read operations * Improving **query efficiency and pagination strategies** * Enhancing **monitoring and alerting for replication lag** * Evaluating **database upgrades and scaling improvements**

## **Summary** _April 13, 2026, FME metrics impact calculations experienced calculations not updated. The root cause was determined to be due to a bug introduced in the software upgrade / release process._ ## **Root Cause** _An internal library upgrade included in the release caused a runtime issue is a legacy execution pathway._ ## **Impact** Feature flag metrics impact calculations were not updating. This issue does not impact experiment calculation, and there is no data loss. ## **Mitigation** _To mitigate we immediately rolled back the update._ ## **Action Items** To prevent such issues from happening again, we are working on fixing gaps in monitoring and alerting for the metrics impact calculations flow.

## **Summary** On April 8, 2026, customers in certain production environments experienced degraded performance and intermittent failures while accessing the platform. This impacted login functionality and execution of new and existing tasks. ## **Root Cause** A spike in internal task processing caused excessive load on the service, leading to resource exhaustion and degraded performance across multiple service instances. ## **Impact** Customers in affected environments experienced: * Slowness and failures during login * Inability to start new tasks in some cases * Failures in ongoing executions ## **Remediation** ‌ **Immediate:** Stabilized the system by resetting affected components and restoring service capacity, which allowed the platform to recover. **Permanent:** Introduced safeguards to limit resource-intensive operations and prevent unbounded processing under high load conditions. ## **Action Items** To prevent such issues from happening again, Harness will * Add limits to high-volume internal processing paths * Audit and enforce safeguards across similar workflows * Improve system resilience under burst load scenarios * Enhance monitoring to detect abnormal load patterns earlier

## **Summary** On April 8, 2026, customers in Prod1 and Prod2 experienced degraded performance when logging into the Harness platform. Additionally, in Prod2, customers were unable to start new pipeline executions and some running pipelines failed. The issue lasted approximately 1 hour and 35 minutes. ## **Root Cause** The issue was caused by a sudden surge of task reassignment requests triggered after customer delegate restarts. This resulted in a high volume of backend processing requests that exceeded expected limits, leading to elevated resource utilization and degraded performance of the Harness Manager service. ## **Impact** * Customers in Prod1 and Prod2 experienced login failures and degraded user operations. * Customers in Prod2 were unable to start new pipeline executions, and some ongoing executions failed. * All customers in the affected clusters experienced service degradation during the incident window. ## **Remediation** **Immediate:** * Restarted affected services and stabilized system performance, restoring login and pipeline functionality. **Permanent:** * Introduced safeguards to limit backend processing for large task reassignment scenarios. * Identifying and applying limits to similar high-volume operations to prevent resource exhaustion. ## **Action Items** To prevent from such issues from happening again * Implement query limits for high-volume task processing scenarios. * Audit and enforce limits across similar backend operations so that we can be resilient. * Enhance monitoring and alerting for abnormal spikes in task reassignment and resource utilization.

## **Summary** On April 7, 2026, customers using the Internal Developer Portal \(IDP\) in certain production environments experienced service disruption where the IDP UI became inaccessible. Users encountered errors when attempting to access the module. ## **Root Cause** A configuration change introduced during a routine deployment prevented the system from correctly routing incoming requests to the IDP service, resulting in loss of access to the UI. ## **Impact** Customers using IDP in affected environments were unable to access the portal UI during the incident window. Other modules and environments remained unaffected. ## **Remediation** **Immediate:** Rolled back the recent configuration change and restored service routing, which recovered access to the IDP module. **Permanent:** Implemented additional safeguards in the deployment process to validate configuration changes and ensure compatibility before rollout. ## **Action Items** To prevent such issues from happening again, we are taking the following steps. * Enhancing our release process and UI validation * Improve monitoring and alerting for early detection of routing issues

## **Summary** On April 2, 2026, customers experienced failures in CI pipelines during S3 upload steps following a routine delegate upgrade. The issue primarily impacted customers using cross-account AWS role assumption with inherit-from-delegate connectors. ## **Impact** Few customers across Prod1 and Prod2 using CI pipelines using S3 upload with cross-account role assumption experienced Artifact uploads failures, blocking downstream deployments ## **Root Cause** A change introduced during the delegate upgrade altered how AWS credentials were passed to CI steps. This resulted in **partial credentials being provided to the S3 upload plugin**, which triggered a latent issue in the plugin’s credential selection logic. Instead of executing the intended cross-account role assumption flow, the plugin attempted authentication using incomplete credentials, leading to failures. ## **Mitigation** * Rolled back delegate to the previous stable version * Restored original credential handling behavior * Service functionality recovered immediately after rollback ## **Next Steps** To prevent such issues from happening again we will: * Improve validation of credential handling in CI steps * Expand automated test coverage for cross-account scenarios * Reintroduce changes behind proper feature flags with full end-to-end testing

The issue was caused by elevated latency from AWS in the affected region, impacting operations such as warm-up, cool-down, schedule execution, and traffic detection. We have now isolated this region to prevent impact on other customers. Resources in me-south-1 will continue to experience the issue until the region fully recovers. We are actively monitoring the situation and will provide further updates as available.

## **Summary** On March 25, 2026, between approximately **3:30 PM and 6:00 PM IST**, the STO service in the **Prod1 environment** experienced **intermittent failures** while processing scan uploads. This resulted in **step failures for some pipeline executions** during the incident window. ## **Root Cause** During a scheduled internal data backfill activity, the STO service experienced **increased database load**. Concurrently, a recent change in the scan upload processing path introduced additional latency under these conditions. The combination of elevated load and increased query execution time caused some scan upload requests to exceed processing thresholds and fail. Retry attempts further amplified system load, leading to intermittent failures. ## **Impact** * Intermittent **scan upload failures \(500 errors\)** during pipeline execution * Some pipelines experienced **step failures or delays due to retries** * No impact to previously uploaded scan results or other STO functionality ## **Mitigation/Remediation** ### **Immediate** * Stopped the internal backfill activity to reduce database load * Optimized the scan upload processing query ### **Permanent** * Introduced safeguards for background jobs to prevent impact on production workloads * Improved performance of critical database paths * Enhanced monitoring to detect abnormal load and retry amplification earlier ## **Action Items** To prevent such issues from happening again: * Implement throttling and isolation for background/backfill jobs * Add protections for critical request paths under load * Improve alerting on database latency and retry patterns * Strengthen validation for production-like load conditions