Harness Outage History

Started Apr 28, 2026, 03:28 PM UTC· resolved Apr 28, 2026, 07:03 PM UTC

1. investigating Apr 28, 2026, 03:28 PM UTC

   We are currently investigating this issue.

Read the full incident report →

Started Apr 28, 2026, 11:51 AM UTC· resolved Apr 28, 2026, 02:16 PM UTC

1. investigating Apr 28, 2026, 11:51 AM UTC

   We are currently investigating this issue.

2. monitoring Apr 28, 2026, 12:14 PM UTC

   A fix has been implemented and we are monitoring the results.

Read the full incident report →

Started Apr 27, 2026, 08:00 PM UTC· resolved Apr 27, 2026, 08:00 PM UTC

1. resolved Apr 27, 2026, 10:27 PM UTC

   We were seeing slowness while executing pipelines

2. postmortem Apr 29, 2026, 07:53 PM UTC

   ## **Summary** On April 27, 2026, customers running pipelines in the Prod3 environment experienced intermittent slowness in pipeline execution and delays in execution status updates in the UI. It was caused by a unexpected spike causing contention on a backend database supporting pipeline orchestration. The issue was mitigated and fully resolved. ## **Impact** **Incident window:** April 27, 2026, 1:00 PM – 3:12 PM PDT * Pipeline executions ran slower than normal; some executions took longer than expected to complete. For pipelines with stricter timeouts, there could be failures. * No widespread pipeline failures were observed * Execution view in the UI lagged behind real-time pipeline progress There was no data loss. The majority of pipelines continued to execute successfully, with the primary impact being increased latency and delayed UI updates. ## **Root Cause** Pipeline orchestration relies on a backend database to track execution state and power the execution view in the UI. During the incident, we had a spike of load, leading to increased query latency across the orchestration layer.This resulted in a backlog, causing UI updates to lag behind actual pipeline execution until the system was scaled. ## **Remediation** **Immediate Mitigation** * Scaled up the affected database instance to increase CPU capacity * Reduced query latency and eliminated lock contention * Cleared the execution-view update backlog within ~30 minutes These actions restored normal pipeline performance and UI responsiveness. ## **Action Items** To prevent such issues from happening again. * **Capacity Improvements:**Updated Prod3 capacity baseline to prevent similar resource constraints * **Proactive Detection:** Enhancing monitoring and alerting for backend resource utilization, lock contention, and critical query latency

Read the full incident report →

Started Apr 27, 2026, 04:05 PM UTC· resolved Apr 27, 2026, 04:24 PM UTC

1. investigating Apr 27, 2026, 04:05 PM UTC

   We are currently investigating this issue.

2. identified Apr 27, 2026, 04:08 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Apr 27, 2026, 04:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 24, 2026, 04:06 PM UTC· resolved Apr 24, 2026, 06:59 PM UTC

1. investigating Apr 24, 2026, 07:16 PM UTC

   We are currently investigating this issue.

2. monitoring Apr 24, 2026, 07:29 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 24, 2026, 08:01 PM UTC

   This incident has been resolved.

4. postmortem Apr 30, 2026, 07:16 PM UTC

   ### Summary On April 24, 2026, a large non-batched bulk DELETE operation on the prod-2 primary database triggered lock contention, causing Feature Flag API latency and hung queries across multiple customer SDKs. ### Impact 1. Slow SDK auth/init — SDKs took longer than expected to complete evaluations 2. Elevated latency across many FF APIs 3. Limited to Feature Flag module, prod-2 4. No Data loss ### Root Cause A background cleanup job executed a non-batched, single-transaction delete causing lock contention and API latency spikes **Mitigation** Immediately terminated the offending queries. ### Next Steps / Action Items To prevent such issues from happening again. we are working on 1. Enhanced alerting and observability on long running queries. 2. Permanently replace large single-transaction delete pattern with smaller batched deletes

Read the full incident report →

Started Apr 22, 2026, 09:35 PM UTC· resolved Apr 22, 2026, 09:35 PM UTC

1. resolved May 08, 2026, 05:08 PM UTC

   On April 22, 2026, couple of customers experienced feature flags returning default treatments due to invalid empty segment membership responses being cached and served to SDKs. The issue was isolated to couple of customers under specific high-concurrency conditions involving Large Segments.

2. postmortem May 08, 2026, 05:09 PM UTC

   ## Summary On April 22, 2026, a couple of customer environments experienced feature flags returning default treatments due to invalid empty segment membership responses being cached and served to SDKs. The issue was isolated to couple of customers under specific high-concurrency conditions involving Large Segments. ## Impact * Feature flag evaluations returned default \(fallback\) values for affected keys * Limited to a single customer environment with ~5.5M keys; approximately 0.9% of keys were affected * No other customers or environments were impacted * The condition was triggered by a traffic spike associated with Large Segment updates ‌ ## Root Cause Under high-concurrency conditions triggered by Large Segment updates, an internal rate-limiting mechanism produced responses that were incorrectly interpreted by the data retrieval layer as valid empty results. These empty responses were then cached and served to SDKs, causing affected keys to evaluate with default treatments instead of their configured values. The condition could not occur under normal traffic patterns and required a specific combination of Large Segment updates generating concurrent requests exceeding internal thresholds. ## Remediation **Fix deployed:** Updated the data retrieval path to correctly identify and reject invalid responses from the rate-limiting mechanism, preventing empty data from being cached regardless of traffic conditions. The fix was validated under load testing that reproduced the original failure. ## Action Items * **Recurring load testing in production:** Implementing periodic load tests in a controlled production environment that exhaust service capacity to replicate this class of failure. All keys in the controlled environment will be continuously validated, enabling early detection of data integrity degradation before it reaches customer environments.

Read the full incident report →

Started Apr 19, 2026, 02:09 PM UTC· resolved Apr 19, 2026, 03:24 PM UTC

1. investigating Apr 19, 2026, 02:09 PM UTC

   We are currently investigating this issue.

2. identified Apr 19, 2026, 02:20 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 19, 2026, 03:17 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 19, 2026, 03:24 PM UTC

   This incident has been resolved. Customers using a pinned version older than plugins/harness_terraform:0.214.0 should update to the latest version by following the https://developer.harness.io/docs/continuous-integration/use-ci/set-up-build-infrastructure/harness-ci/#specify-the-harness-ci-images-used-in-your-pipelines. If you are not pinning a specific version, no action is required — your pipelines are already using the updated image.

5. postmortem Apr 30, 2026, 07:04 PM UTC

   ## **Summary** On April 19, 2026, Terraform-based IaCM pipelines failed across production environments due to an issue with Terraform binary verification during runtime. The issue was caused by an expired OpenPGP signing key in a third-party library used to validate Terraform downloads. This resulted in failures when pipelines attempted to install Terraform dynamically. ## **Impact** * Terraform-based IaCM pipelines failed during execution * Failures occurred at runtime when attempting to download/verify Terraform binaries * **Customers Unaffected:** * OpenTofu-based pipelines * Pipelines using pre-installed or cached Terraform binaries Customers pinning plugin versions older than the fixed release continued to experience failures until upgraded. ## **Root Cause** The IaCM Terraform plugin relies on a third-party library \(HashiCorp’s `hc-install`\) to download and verify Terraform binaries. * The library contained a **hardcoded OpenPGP signing key** * This key **expired**, causing verification failures during Terraform installation * HashiCorp had not yet released an updated version with a renewed key ## **Remediation** ### **Immediate Mitigation** * Released **IaCM Terraform plugin v0.214.0** * Modified behavior to: * **Bypass the expired signature verification step** * Continue secure downloads over HTTPS ### **Resolution** * Rolled out the fix across **prod0–prod4** * Pipeline execution functionality was restored ## **Customer Actions Required** * Customers using pinned plugin versions **older than v0.214.0** must: * **Upgrade to v0.214.0 or later** * No action required for customers using default/latest plugin versions ## **Prevention & Next Steps** We are implementing the following improvements: * **Dependency Monitoring** * Proactive monitoring for third-party certificate/key expirations * **Upstream Coordination** * Track HashiCorp release for updated signing key * Re-enable signature verification once available * **Customer Communication** * Notify customers using older pinned versions * **Operational Improvements** * Enhance validation of external dependencies in runtime workflows

Read the full incident report →

Started Apr 16, 2026, 07:15 AM UTC· resolved Apr 16, 2026, 12:02 PM UTC

1. investigating Apr 16, 2026, 10:06 AM UTC

   We are currently investigating this issue.

2. identified Apr 16, 2026, 10:14 AM UTC

   The issue has been identified and a fix is being implemented.

3. identified Apr 16, 2026, 11:08 AM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Apr 16, 2026, 11:41 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Apr 16, 2026, 12:02 PM UTC

   This incident has been resolved.

6. postmortem Apr 23, 2026, 03:34 PM UTC

   On April 16, 2026, Hosted CI Linux pipelines experienced intermittent initialization failures due to an upstream outage affecting package repositories used during environment setup. ### **Impact** A subset of customers running Hosted CI pipelines encountered failures during the initialization phase, preventing jobs from starting successfully. * Affected: 24 accounts * Failed executions: 129 pipelines * Impact duration: ~5 hours 49 minutes ### **Root Cause** The issue was caused by a service disruption in an external package repository provider. During CI environment provisioning, dependency installation requests to this upstream service timed out, causing initialization failures. ### **Remediation** **Immediate Mitigation** We updated runner configurations to bypass dependency installation during initialization and rolled out updated environments across affected clusters. **Permanent Fix** We have improved resilience in our CI infrastructure by: * Using pre-configured environments with required dependencies pre-installed * Eliminating runtime dependency on external package repositories during initialization * Enhancing failure handling for external dependency timeouts ### **Action Items / Next Steps** * Continue improving isolation from external dependencies during environment startup * Strengthen monitoring and alerting for upstream service degradation * Optimize rollout speed for infrastructure changes to reduce mitigation time

Read the full incident report →

Started Apr 15, 2026, 11:55 PM UTC· resolved Apr 16, 2026, 04:15 AM UTC

1. investigating Apr 15, 2026, 11:55 PM UTC

   We are currently investigating this issue.

2. investigating Apr 16, 2026, 12:01 AM UTC

   We are continuing to investigate this issue.

3. monitoring Apr 16, 2026, 04:04 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 16, 2026, 06:05 PM UTC

   This incident has been resolved.

5. postmortem Apr 30, 2026, 05:34 PM UTC

   ## **Summary** On April 15, 2026, between approximately 23:21 UTC and 01:58 UTC, customers using Feature Flag in the prod2 environment experienced delays in feature flag updates. Feature flag changes made via UI or API were successfully processed but were **not immediately reflected**, causing stale flag values to be served. ## **Impact** * **Scope:** Customers on **prod2 environment only** * **Customer Impact:** * Feature flag updates were **delayed or appeared ineffective** * Applications continued serving **stale configurations** * **Other Environments:** No impact to prod0, prod1, or other regions ## **Root Cause** The issue was caused by replication lag in the read replica database used for serving feature flag reads. A **long-running read query** on the replica blocked replication updates from the primary database. This caused a delay in propagating recent feature flag changes to read queries ### **What triggered the issue** * A high-volume API usage pattern involving **large paginated queries on target data.** These queries became **resource-**intensive impacting the database. ## **Mitigation** ### **Immediate Actions** * Identified and **terminated long-running queries** on the replica * Replication resumed and flag updates began reflecting correctly ## **Prevention & Next Steps** We are continuing to strengthen reliability through: * We configured replica to **automatically cancel queries** that block replication beyond a threshold and tuned **query timeouts** for heavy read operations * Improving **query efficiency and pagination strategies** * Enhancing **monitoring and alerting for replication lag** * Evaluating **database upgrades and scaling improvements**

Read the full incident report →

Started Apr 13, 2026, 04:47 PM UTC· resolved Apr 13, 2026, 05:31 PM UTC

1. monitoring Apr 13, 2026, 04:47 PM UTC

   Feature flag metrics impact calculations were not updating. This issue does not impact experiment calculation. Fix is being rolled out. There is no data loss.

2. resolved Apr 13, 2026, 05:31 PM UTC

   After monitoring post fixing the issue, all systems are back to normal and processing metrics impact calculations regularly. We will provide an RCA soon.

3. postmortem Apr 20, 2026, 03:56 PM UTC

   ## **Summary** _April 13, 2026, FME metrics impact calculations experienced calculations not updated. The root cause was determined to be due to a bug introduced in the software upgrade / release process._ ## **Root Cause** _An internal library upgrade included in the release caused a runtime issue is a legacy execution pathway._ ## **Impact** Feature flag metrics impact calculations were not updating. This issue does not impact experiment calculation, and there is no data loss. ## **Mitigation** _To mitigate we immediately rolled back the update._ ## **Action Items** To prevent such issues from happening again, we are working on fixing gaps in monitoring and alerting for the metrics impact calculations flow.

Read the full incident report →

Started Apr 09, 2026, 05:30 AM UTC· resolved Apr 09, 2026, 10:48 AM UTC

1. investigating Apr 09, 2026, 08:34 AM UTC

   Some of the legacy run test step connectivity to test intel service is failing intermittently. We are currently investigating the issue here.

2. identified Apr 09, 2026, 09:38 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 09, 2026, 10:38 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 09, 2026, 10:48 AM UTC

   This incident has been resolved.

5. postmortem Apr 21, 2026, 01:18 PM UTC

   ## **Summary** On April 8, 2026, customers in certain production environments experienced degraded performance and intermittent failures while accessing the platform. This impacted login functionality and execution of new and existing tasks. ## **Root Cause** A spike in internal task processing caused excessive load on the service, leading to resource exhaustion and degraded performance across multiple service instances. ## **Impact** Customers in affected environments experienced: * Slowness and failures during login * Inability to start new tasks in some cases * Failures in ongoing executions ## **Remediation** ‌ **Immediate:** Stabilized the system by resetting affected components and restoring service capacity, which allowed the platform to recover. **Permanent:** Introduced safeguards to limit resource-intensive operations and prevent unbounded processing under high load conditions. ## **Action Items** To prevent such issues from happening again, Harness will * Add limits to high-volume internal processing paths * Audit and enforce safeguards across similar workflows * Improve system resilience under burst load scenarios * Enhance monitoring to detect abnormal load patterns earlier

Read the full incident report →

Started Apr 09, 2026, 01:05 AM UTC· resolved Apr 09, 2026, 02:42 AM UTC

1. investigating Apr 09, 2026, 01:43 AM UTC

   We are currently investigating this issue.

2. identified Apr 09, 2026, 02:02 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 09, 2026, 02:33 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 09, 2026, 02:42 AM UTC

   This incident has been resolved.

5. postmortem Apr 16, 2026, 09:22 PM UTC

   ## **Summary** On April 8, 2026, customers in Prod1 and Prod2 experienced degraded performance when logging into the Harness platform. Additionally, in Prod2, customers were unable to start new pipeline executions and some running pipelines failed. The issue lasted approximately 1 hour and 35 minutes. ## **Root Cause** The issue was caused by a sudden surge of task reassignment requests triggered after customer delegate restarts. This resulted in a high volume of backend processing requests that exceeded expected limits, leading to elevated resource utilization and degraded performance of the Harness Manager service. ## **Impact** * Customers in Prod1 and Prod2 experienced login failures and degraded user operations. * Customers in Prod2 were unable to start new pipeline executions, and some ongoing executions failed. * All customers in the affected clusters experienced service degradation during the incident window. ## **Remediation** **Immediate:** * Restarted affected services and stabilized system performance, restoring login and pipeline functionality. **Permanent:** * Introduced safeguards to limit backend processing for large task reassignment scenarios. * Identifying and applying limits to similar high-volume operations to prevent resource exhaustion. ## **Action Items** To prevent from such issues from happening again * Implement query limits for high-volume task processing scenarios. * Audit and enforce limits across similar backend operations so that we can be resilient. * Enhance monitoring and alerting for abnormal spikes in task reassignment and resource utilization.

Read the full incident report →

Started Apr 08, 2026, 01:09 PM UTC· resolved Apr 08, 2026, 03:00 PM UTC

1. investigating Apr 08, 2026, 01:09 PM UTC

   We are currently investigating this issue.

2. monitoring Apr 08, 2026, 01:54 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 08, 2026, 07:44 PM UTC

   This incident has been resolved.

4. postmortem Apr 30, 2026, 07:17 PM UTC

   .

Read the full incident report →

Started Apr 07, 2026, 10:54 AM UTC· resolved Apr 07, 2026, 11:08 AM UTC

1. monitoring Apr 07, 2026, 10:54 AM UTC

   A fix has been implemented and we are monitoring the results.

2. monitoring Apr 07, 2026, 10:59 AM UTC

   We are continuing to monitor for any further issues.

3. resolved Apr 07, 2026, 11:08 AM UTC

   This incident has been resolved.

4. postmortem Apr 21, 2026, 12:58 PM UTC

   ## **Summary** On April 7, 2026, customers using the Internal Developer Portal \(IDP\) in certain production environments experienced service disruption where the IDP UI became inaccessible. Users encountered errors when attempting to access the module. ## **Root Cause** A configuration change introduced during a routine deployment prevented the system from correctly routing incoming requests to the IDP service, resulting in loss of access to the UI. ## **Impact** Customers using IDP in affected environments were unable to access the portal UI during the incident window. Other modules and environments remained unaffected. ## **Remediation** **Immediate:** Rolled back the recent configuration change and restored service routing, which recovered access to the IDP module. **Permanent:** Implemented additional safeguards in the deployment process to validate configuration changes and ensure compatibility before rollout. ## **Action Items** To prevent such issues from happening again, we are taking the following steps. * Enhancing our release process and UI validation * Improve monitoring and alerting for early detection of routing issues

Read the full incident report →

Started Apr 03, 2026, 04:53 PM UTC· resolved Apr 04, 2026, 01:59 PM UTC

1. investigating Apr 03, 2026, 04:53 PM UTC

   We are currently investigating this issue.

2. identified Apr 03, 2026, 04:59 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 03, 2026, 09:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Apr 03, 2026, 09:30 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Apr 04, 2026, 01:59 PM UTC

   This incident has been resolved.

6. postmortem Jun 24, 2026, 12:42 AM UTC

   .

Read the full incident report →

Started Apr 02, 2026, 03:13 PM UTC· resolved Apr 02, 2026, 05:53 PM UTC

1. investigating Apr 02, 2026, 03:13 PM UTC

   We are investigating a degradation in CI steps when using AWS connectors and inherited authentication.

2. resolved Apr 02, 2026, 05:53 PM UTC

   This incident has been resolved.

3. postmortem Apr 17, 2026, 03:29 PM UTC

   ## **Summary** On April 2, 2026, customers experienced failures in CI pipelines during S3 upload steps following a routine delegate upgrade. The issue primarily impacted customers using cross-account AWS role assumption with inherit-from-delegate connectors. ## **Impact** Few customers across Prod1 and Prod2 using CI pipelines using S3 upload with cross-account role assumption experienced Artifact uploads failures, blocking downstream deployments ## **Root Cause** A change introduced during the delegate upgrade altered how AWS credentials were passed to CI steps. This resulted in **partial credentials being provided to the S3 upload plugin**, which triggered a latent issue in the plugin’s credential selection logic. Instead of executing the intended cross-account role assumption flow, the plugin attempted authentication using incomplete credentials, leading to failures. ## **Mitigation** * Rolled back delegate to the previous stable version * Restored original credential handling behavior * Service functionality recovered immediately after rollback ## **Next Steps** To prevent such issues from happening again we will: * Improve validation of credential handling in CI steps * Expand automated test coverage for cross-account scenarios * Reintroduce changes behind proper feature flags with full end-to-end testing

Read the full incident report →

Started Apr 01, 2026, 06:44 AM UTC· resolved Apr 02, 2026, 08:40 AM UTC

1. identified Apr 01, 2026, 06:16 AM UTC

   The issue has been identified and a fix is being implemented.

2. identified Apr 01, 2026, 06:44 AM UTC

   Status update : CCM AutoStopping functionality for the AWS cloud provider is currently impacted due to increased latency from AWS in the me-south-1 region. This is affecting multiple operations, including warm-up, cool-down, schedule execution, and traffic detection. In addition, CCM Asset Governance functionality is also impacted for resources in the me-south-1 region. We are actively working on isolating/excluding the affected region to restore functionality for the remaining customers. Resources within the me-south-1 region may continue to experience issues until the region fully recovers.

3. monitoring Apr 01, 2026, 08:20 AM UTC

   Update: AutoStopping functionality for AWS has been restored for all regions except me-south-1. The issue was caused by elevated latency from AWS in the affected region, impacting operations such as warm-up, cool-down, schedule execution, and traffic detection. We have now isolated this region to prevent impact on other customers. Resources in me-south-1 will continue to experience the issue until the region fully recovers. We are actively monitoring the situation and will provide further updates as available.

4. resolved Apr 02, 2026, 08:40 AM UTC

   This incident has been resolved.

5. postmortem Apr 14, 2026, 03:30 AM UTC

   The issue was caused by elevated latency from AWS in the affected region, impacting operations such as warm-up, cool-down, schedule execution, and traffic detection. We have now isolated this region to prevent impact on other customers. Resources in me-south-1 will continue to experience the issue until the region fully recovers. We are actively monitoring the situation and will provide further updates as available.

Read the full incident report →

Started Mar 25, 2026, 10:00 AM UTC· resolved Mar 25, 2026, 12:30 PM UTC

1. investigating Mar 25, 2026, 03:53 PM UTC

   We are currently investigating this issue.

2. resolved Mar 25, 2026, 03:53 PM UTC

   This incident has been resolved.

3. postmortem Mar 26, 2026, 07:53 PM UTC

   ## **Summary** On March 25, 2026, between approximately **3:30 PM and 6:00 PM IST**, the STO service in the **Prod1 environment** experienced **intermittent failures** while processing scan uploads. This resulted in **step failures for some pipeline executions** during the incident window. ## **Root Cause** During a scheduled internal data backfill activity, the STO service experienced **increased database load**. Concurrently, a recent change in the scan upload processing path introduced additional latency under these conditions. The combination of elevated load and increased query execution time caused some scan upload requests to exceed processing thresholds and fail. Retry attempts further amplified system load, leading to intermittent failures. ## **Impact** * Intermittent **scan upload failures \(500 errors\)** during pipeline execution * Some pipelines experienced **step failures or delays due to retries** * No impact to previously uploaded scan results or other STO functionality ## **Mitigation/Remediation** ### **Immediate** * Stopped the internal backfill activity to reduce database load * Optimized the scan upload processing query ### **Permanent** * Introduced safeguards for background jobs to prevent impact on production workloads * Improved performance of critical database paths * Enhanced monitoring to detect abnormal load and retry amplification earlier ## **Action Items** To prevent such issues from happening again: * Implement throttling and isolation for background/backfill jobs * Add protections for critical request paths under load * Improve alerting on database latency and retry patterns * Strengthen validation for production-like load conditions

Read the full incident report →

Started Mar 24, 2026, 06:14 PM UTC· resolved Mar 24, 2026, 08:37 PM UTC

1. investigating Mar 24, 2026, 06:14 PM UTC

   We are currently investigating this issue.

2. identified Mar 24, 2026, 06:25 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 24, 2026, 07:03 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Mar 24, 2026, 07:03 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Mar 24, 2026, 08:37 PM UTC

   This incident has been resolved.

6. postmortem Mar 31, 2026, 03:31 AM UTC

   ## Summary Between **05:48 PM UTC on March 18** and **04:58 PM UTC on March 19**, two customers using [app.split.io](http://app.split.io/) with **SAML authentication using SCIM** experienced login failures. The issue was limited to customers with specific identity provisioning configurations. ## Root Cause A service update was deployed to a login-related component that changed internal request handling behavior. This caused authentication workflows for SAML-configured customers with identity provisioning enabled to fail during login processing. ## Impact * 2 customers using SAML authentication **with SCIM provisioning enabled** were unable to log in to [app.split.io](http://app.split.io/). * Customers using SAML authentication without SCIM were not affected and could log in normally. * Username and password login was unaffected. * SDKs, feature flags, and experiments continued functioning normally. * No data loss occurred. ## Remediation * Rolled back the service update that introduced the issue. * Restored SAML login for affected customers. ## Action Items To prevent such issues from happening again, we are doing/completed the following: * **Improve detection capability**: Introduced a high-priority alert to detect authentication failures sooner, enabling faster response if similar issues occur in the future. * **Reduce legacy surface area**: Continue guiding remaining customers toward the Harness Platform, where authentication flows are unified and improvements can be delivered more consistently over time.

Read the full incident report →

Started Mar 19, 2026, 04:14 PM UTC· resolved Mar 19, 2026, 04:59 PM UTC

1. investigating Mar 19, 2026, 04:14 PM UTC

   The issue has been identified and a fix is in place.

2. monitoring Mar 19, 2026, 04:42 PM UTC

   The issue has been identified and a fix is in place. (Note: It is still in a degraded state)

3. monitoring Mar 19, 2026, 04:42 PM UTC

   The issue has been identified and a fix is in place.

4. monitoring Mar 19, 2026, 04:59 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Mar 19, 2026, 04:59 PM UTC

   This incident has been resolved.

6. postmortem Apr 02, 2026, 12:45 AM UTC

   ## Summary Between **05:48 PM UTC on March 18** and **04:58 PM UTC on March 19**, two customers using [app.split.io](http://app.split.io/) with **SAML authentication using SCIM** experienced login failures. The issue was limited to customers with specific identity provisioning configurations. ## Root Cause A service update was deployed to a login-related component that changed internal request handling behavior. This caused authentication workflows for SAML-configured customers with identity provisioning enabled to fail during login processing. ## Impact * 2 customers using SAML authentication **with SCIM provisioning enabled** were unable to log in to [app.split.io](http://app.split.io/). * Customers using SAML authentication without SCIM were not affected and could log in normally. * Username and password login was unaffected. * SDKs, feature flags, and experiments continued functioning normally. * No data loss occurred. ## Remediation * Rolled back the service update that introduced the issue. * Restored SAML login for affected customers. ## Action Items To prevent such issues from happening again, here are the action items. * **Improve detection capability**: Introduced a high-priority alert to detect authentication failures sooner, enabling faster response if similar issues occur in the future. * **Reduce legacy surface area**: Continue guiding remaining customers toward the Harness Platform, where authentication flows are unified and improvements can be delivered more consistently over time.

Read the full incident report →

Started Mar 17, 2026, 06:04 PM UTC· resolved Mar 17, 2026, 06:39 PM UTC

1. investigating Mar 17, 2026, 06:04 PM UTC

   We are currently investigating this issue.

2. identified Mar 17, 2026, 06:07 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 17, 2026, 06:10 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 17, 2026, 06:39 PM UTC

   This incident has been resolved.

5. postmortem Mar 24, 2026, 07:17 PM UTC

   ## **Summary** On March 17, between approximately **3:04 PM and 4:05 PM PST**, customers experienced **degraded performance and intermittent unavailability** for Test Intelligence \(TI\) related APIs. This impacted test result uploads and report access. ## **Root Cause** A recent change introduced additional background processing during test result ingestion, which significantly increased database load. This led to resource saturation and caused elevated latency and temporary service disruption. ## **Impact** * **Degraded performance** for test uploads and report retrieval * **Intermittent API unavailability \(~7 minutes\)** * Affected customers experienced slower response times across TI-related workflows ## **Remediation** ### **Immediate** * Increased database capacity to stabilize performance * Applied configuration changes to reduce system load ### **Permanent** * Optimized processing logic to reduce database overhead * Introduced safeguards to prevent similar load amplification scenarios ## **Action Items** * Improve performance validation using production-scale datasets * Enhance safeguards for new feature rollouts * Strengthen monitoring to detect abnormal load patterns earlier * Add controls to limit impact scope of similar changes

Read the full incident report →

Started Mar 17, 2026, 05:05 PM UTC· resolved Mar 17, 2026, 07:33 PM UTC

1. investigating Mar 17, 2026, 04:36 PM UTC

   The issue started around ~6:45AM PT and the team is currently investigating

2. monitoring Mar 17, 2026, 05:05 PM UTC

   We are now monitoring the results.

3. resolved Mar 17, 2026, 11:38 PM UTC

   This incident has been resolved.

4. postmortem Mar 18, 2026, 10:57 PM UTC

   ## **Summary** _March 17, 2026, FME events & impressions ingestion experienced significant degradation, resulting in elevated latency and error rates. The impact was traced to degraded performance in the underlying shared infrastructure used for event processing._ ## **Root Cause** _We had an unexpected surge in traffic which caused stress on our systems._ ## **Impact** SDKs sending impressions and events would experience elevated error logging and continue to retry, with differing policies depending on the particular SDK and it’s retry policy, which are designed and tailored to each runtime environment to avoid any application impact. In some scenarios, events and impressions may be lost if they are not successfully delivered according to the SDK’s specific retry policy. There was no impact to our control plane services and feature flag delivery and evaluations continued to work without any disruption. ## **Mitigation** _To mitigate, we immediately increased capacity to handle the bursty traffic._ ## **Action Items** To prevent such issues from happening again, we are working on the following items: 1. Evaluate and enforce per customer rate-limit . 2. _Improve the auto-scaling and on-demand network infrastructure scale up._ 3. _Improve resiliency of the ingestion layer_ _._

Read the full incident report →

Started Mar 13, 2026, 04:12 PM UTC· resolved Mar 13, 2026, 05:34 PM UTC

1. investigating Mar 13, 2026, 04:12 PM UTC

   We are noticing degraded performance in CI Steps in Prod 2 and Prod 3 environments The issue is intermittent. We are investigating the cause

2. identified Mar 13, 2026, 04:16 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 13, 2026, 04:27 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 13, 2026, 05:34 PM UTC

   This incident has been resolved.

5. postmortem Mar 17, 2026, 11:54 PM UTC

   ### **Summary** On March 13, 2026, customers running CI pipelines in the Prod2 and Prod3 environments experienced **slower-than-normal CI step execution times**. Investigation showed that CI steps were delayed due to a backlog in internal response processing within the CI Manager. While individual plugin steps executed normally, their completion notifications were delayed, causing pipeline stages to appear significantly slower. The issue began around **2:00 AM PT** and affected some customers until mitigation actions were applied. ### **Root Cause** The delay was caused by a **backlog in the CI response processing pipeline** A combination of factors contributed to the backlog: * A brief latency spike affecting internal services, including a DB query executed by the pipeline service. * Increased response processing load that caused iterator workers to stall while waiting on shared resources. These conditions caused CI Manager to accumulate pending responses, which delayed the reporting of step completion even though the underlying plugin execution completed quickly. ### **Impact** Customers experienced **significantly increased CI pipeline step durations**, even though the actual execution time of the steps was minimal. Impact included: * CI pipeline stages appearing to take **longer than expected** * Slower overall pipeline execution times for some customers in **Prod2 and Prod3** * No data loss or failed builds occurred Pipeline performance returned to normal after mitigation actions were applied. ### **Mitigation** **Immediate** * Restarted CI Manager components in affected environments to clear the response processing backlog. * Verified CI pipeline execution times returned to baseline levels. **Permanent** * Implemented new monitoring and alerting on CI iterator response processing latency. * Introduced proactive detection thresholds to identify abnormal processing delays earlier. ### **Action Items** To prevent such issues from happening again we are taking the following steps: * Improve monitoring for CI response processing latency to detect backlog formation sooner. * Investigate and optimize query behavior associated with pipeline-service reads. * Review CI Manager response processing design to reduce sensitivity to latency spikes. * Add safeguards to prevent iterator workers from entering non-recovering states during transient spikes.

Read the full incident report →

Started Mar 12, 2026, 04:20 PM UTC· resolved Mar 12, 2026, 07:16 PM UTC

1. investigating Mar 12, 2026, 04:20 PM UTC

   Customers may continue to see that some pipeline executions show that they are "running" even though they have completed, aborted, or failed as a result of yesterday's incident. (https://status.harness.io/incidents/4y4dl47v2qhc) This behavior is a UI-only artifact from the incident and should not affect customers' ability to start new executions. We are working on clearing these artifacts.

2. identified Mar 12, 2026, 04:21 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Mar 12, 2026, 07:16 PM UTC

   This incident has been resolved.

4. postmortem Mar 17, 2026, 11:43 PM UTC

   ### **Summary** On March 11, 2026, customers experienced pipeline failures and degraded UI performance\(incorrect status of states\) in the Prod2 environment. The issue was caused by a degradation in an internal shared infrastructure component used for coordination across services. The incident began around **7:10 AM PST** and was fully mitigated by approximately **10:12 AM PST**. During this period, pipeline execution throughput was significantly impacted for affected customers. ### **Root Cause** The issue was caused by resource saturation in a shared infrastructure component used for distributed coordination, which led to increased latency and failures in service-to-service communication. As a result, pipeline execution services were unable to process workloads efficiently, leading to a buildup of queued tasks and reduced system throughput. ### **Impact** Customers experienced the following: * Pipeline executions failing or not progressing * Increased pipeline execution times * UI delays due to processing backlogs The impact was limited to specific production environments and no data loss occurred. ### **Mitigation** **Immediate** * Redirected services to a higher-capacity infrastructure instance to restore normal processing * Cleared accumulated processing backlogs to recover system throughput * Scaled supporting services to stabilize performance **Permanent** * Improved monitoring and alerting for early detection of resource saturation * Implemented capacity and scaling improvements to handle higher load scenarios * Initiated architectural improvements to reduce reliance on shared coordination components ### **Action Items** To prevent such issues from happening again we are taking several steps: * Enhance alerting to detect early signs of infrastructure saturation * Review and optimize system behavior under high concurrency scenarios * Continue investigation into the triggering conditions and incorporate findings into long-term improvements

Read the full incident report →

Started Mar 11, 2026, 04:20 PM UTC· resolved Mar 11, 2026, 06:45 PM UTC

1. investigating Mar 11, 2026, 03:38 PM UTC

   We are currently investigating this issue.

2. identified Mar 11, 2026, 03:42 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Mar 11, 2026, 03:55 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Mar 11, 2026, 04:20 PM UTC

   Dashboards are in recovering phase. We are continuing to work on a fix for pipelines issue.

5. identified Mar 11, 2026, 05:18 PM UTC

   Pipeline executions are going fine,there is a delay to view it on the UI.

6. identified Mar 11, 2026, 06:06 PM UTC

   Currently all the executions are going on track and will complete. The UI is showing a delayed status. We are currently expediting the UI recovery.

7. monitoring Mar 11, 2026, 06:15 PM UTC

   A fix has been implemented and we are monitoring the results.

8. monitoring Mar 11, 2026, 06:33 PM UTC

   A fix has been implemented and we are monitoring the results.

9. resolved Mar 11, 2026, 06:45 PM UTC

   This incident has been resolved.

10. postmortem Mar 17, 2026, 11:40 PM UTC

    ### **Summary** On March 11, 2026, customers experienced pipeline failures and degraded UI performance\(incorrect status of states\) and CCM Dashboards were not accessible to the affected customers in the Prod2 environment. The issue was caused by a degradation in an internal shared infrastructure component used for coordination across services. The incident began around **7:10 AM PST** and was fully mitigated by approximately **10:12 AM PST**. During this period, pipeline execution throughput was significantly impacted for affected customers. ### **Root Cause** The issue was caused by resource saturation in a shared infrastructure component used for distributed coordination, which led to increased latency and failures in service-to-service communication. As a result, pipeline execution services were unable to process workloads efficiently, leading to a buildup of queued tasks and reduced system throughput. ### **Impact** Customers experienced the following: * Pipeline executions failing or not progressing * Increased pipeline execution times * UI delays due to processing backlogs The impact was limited to specific production environments and no data loss occurred. ### **Mitigation** **Immediate** * Redirected services to a higher-capacity infrastructure instance to restore normal processing * Cleared accumulated processing backlogs to recover system throughput * Scaled supporting services to stabilize performance **Permanent** * Improved monitoring and alerting for early detection of resource saturation * Implemented capacity and scaling improvements to handle higher load scenarios * Initiated architectural improvements to reduce reliance on shared coordination components ### **Action Items** To prevent such issues from happening again we are taking several steps: * Enhance alerting to detect early signs of infrastructure saturation * Review and optimize system behavior under high concurrency scenarios * Continue investigation into the triggering conditions and incorporate findings into long-term improvements

Read the full incident report →