Harness Outage History

Started May 20, 2026, 10:48 AM UTC· resolved May 20, 2026, 12:38 PM UTC

1. investigating May 20, 2026, 10:48 AM UTC

   We have received reports of slowness in Harness pipelines in Prod3 cluster. We are actively investigating this issue. Please monitor this space for further updates.

2. monitoring May 20, 2026, 11:52 AM UTC

   A fix has been implemented which resolved the pipeline slowness , we are currently monitoring the results.

3. resolved May 20, 2026, 12:38 PM UTC

   This incident has been resolved.

4. postmortem Jun 01, 2026, 07:56 AM UTC

   # **Summary** On May 20 and May 21, 2026, customers on the Prod3 cluster experienced degraded pipeline performance on two separate occasions. In both cases, pipeline executions ran slower than usual and execution graph rendering was delayed because database operations were being queued on the backing MongoDB instance. Both incidents were mitigated by adjusting database write settings and adding capacity headroom, which relieved pressure on MongoDB and allowed normal operations to resume. # **Root Cause** A higher-than-usual rate of concurrent updates against a frequently modified document used during pipeline execution produced a high rate of write conflicts in MongoDB. Each conflict triggered an automatic retry, which compounded under load and drove CPU utilization on the database to a sustained high level. As CPU saturated: * MongoDB started queuing incoming commands instead of executing them immediately. * Operations dependent on those writes — pipeline step transitions and execution graph generation — slowed down or stalled while waiting for the queue to drain. # **Customer Impact** Customers on Prod3 were impacted during two windows: * May 20, 2026: ~3:15 AM to ~4:45 AM PST \(~1h 30m\). * May 21, 2026: ~6:42 AM to ~7:47 AM PST \(~1h 5m\). In both windows: * Pipeline executions ran significantly slower, with delays in step initialization and execution graph generation. * Pipelines with tight configured timeouts may have timed out as a result of the slowness. No execution failures were observed beyond timeout-related effects, and no full outage occurred. All Harness modules that rely on Pipelines on Prod3 were affected for the duration of each incident. # **Resolution** After identifying that database contention was the bottleneck, we adjusted the pipeline service's database write settings to reduce overhead on each operation and added additional database capacity to absorb the load. Together these reduced write conflicts and allowed the queued commands to drain, after which pipeline operations returned to normal. # **Prevention and Improvements** 1. Strengthen concurrency control around the high-contention update path so that concurrent updates to the same document no longer race in a way that produces write conflicts at scale. 2. Strengthen load and concurrency testing on hot execution paths so similar contention patterns are caught before they affect production traffic. 3. Improve proactive alerting on database write conflicts, queued commands, and sustained CPU utilization so the team can intervene before customer-facing latency is affected.

Read the full incident report →

Started May 15, 2026, 08:35 PM UTC· resolved May 17, 2026, 05:37 AM UTC

1. investigating May 15, 2026, 08:35 PM UTC

   We are currently investigating this issue.

2. monitoring May 15, 2026, 08:59 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring May 17, 2026, 05:36 AM UTC

   We are continuing to monitor for any further issues.

4. resolved May 17, 2026, 05:37 AM UTC

   This incident has been resolved.

5. postmortem May 28, 2026, 12:18 AM UTC

   # Summary Dashboards service in Prod3 experienced intermittent failures, causing dashboards to return errors and become unavailable to some customers. ‌ ## Root Cause Analysis The incident was caused by slow Looker `search_dashboards` API calls degrading from 7 seconds to over 30 seconds, which saturated the worker thread pool and prevented the health endpoint from responding to Kubernetes liveness and readiness probes. ‌ ## Mitigation Steps Taken ### Immediate Mitigations 1. **Increased liveness probe timeout** from 15s to 30s \(failure threshold kept at 3, allowing up to 90s tolerance\) 2. **Doubled thread count** , increasing total concurrency 3. **Increased pod replica count** to maintain higher minimum availability and reduce risk of all pods becoming simultaneously unavailable ‌ ## Actions Harness will work on the following Action items to prevent recurrence. 1. **Optimize Looker SDK timeout:** Tune timeout to Looker SDK `search_dashboards` calls to prevent indefinite thread occupation 2. **Optimize Looker queries :** Investigate Looker-side query performance degradation to understand why `search_dashboards` latency increased

Read the full incident report →

Started May 15, 2026, 04:54 AM UTC· resolved May 21, 2026, 04:34 PM UTC

1. investigating May 15, 2026, 04:54 AM UTC

   We are currently investigating this issue.

2. monitoring May 15, 2026, 05:14 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring May 15, 2026, 05:23 AM UTC

   We are continuing to monitor for any further issues.

4. resolved May 21, 2026, 04:34 PM UTC

   This incident has been resolved.

5. postmortem May 27, 2026, 11:55 PM UTC

   same as [https://status.harness.io/incidents/pxl6nhjdnlwj](https://status.harness.io/incidents/pxl6nhjdnlwj)

Read the full incident report →

Started May 14, 2026, 01:36 PM UTC· resolved May 14, 2026, 02:10 PM UTC

1. investigating May 14, 2026, 01:36 PM UTC

   We are currently investigating this issue.

2. investigating May 14, 2026, 01:39 PM UTC

   We are continuing to investigate this issue.

3. investigating May 14, 2026, 01:40 PM UTC

   We are continuing to investigate this issue.

4. identified May 14, 2026, 02:03 PM UTC

   The issue has been identified and a fix is being implemented.

5. monitoring May 14, 2026, 02:08 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved May 14, 2026, 02:10 PM UTC

   This incident has been resolved.

7. postmortem May 27, 2026, 11:47 PM UTC

   On May 14, 2026 , some customers running pipelines in the Prod4 production environment observed pipeline create and update requests that were slow or failed, and pipeline stages that did not start, produced no logs, and were eventually auto-aborted as “stuck”. The issue was caused by an underlying compute node in our Prod4 cluster being recycled abruptly. ## **Impact** During the incident window \(approximately 5:38 PM PDT on May 14 to 9:47 PM PDT on May 14, 2026\): * Some pipeline create and update requests on Prod4 were slow or failed. * Some Prod4 pipeline executions hung at the stage-start step, producing no logs, and were eventually auto-aborted as “stuck” after a timeout. * Behavior was intermittent — only pipelines whose requests were routed to an affected service pod were impacted; other pipelines continued to execute normally. There was **no data loss**. The majority of pipelines on Prod4 continued to execute successfully throughout the incident — the primary impact was that affected create/update requests slowed down or failed, and a subset of pipelines could not progress and had to be aborted and re-run after mitigation. Overall service availability was degraded during this window. ## **Root Cause** During the incident, an underlying compute node in our Prod4 cluster was recycled by the cloud provider without completing its normal graceful-drain process, so the supporting-service pods running on that node were terminated abruptly. As a result, in-flight requests from the backend service to those pods were left without a response. ## **Mitigation** Harness completed the following immediate mitigation steps: * Restarted the affected supporting-service pods to restore healthy targets. * Restarted the pipeline service in Prod4 to clear the blocked worker threads. This is what fully restored normal pipeline create/update behavior and stage-start behavior; restarting only the supporting service was not enough on its own. * Confirmed pipeline executions returned to normal and updated the status page to mitigated. These actions restored pipeline execution behavior and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented: * Optimize timeouts to the pipeline service’s plan-creation requests so that when a supporting service goes away unexpectedly, the worker threads recover automatically instead of remaining blocked. * Investigate the abrupt node-recycle behavior in Prod4 with our cloud provider to ensure pods running on a recycled node receive a graceful shutdown signal in the future. * Add proactive paging alerts on service worker-thread saturation, so this failure mode is detected before it becomes a impacting issue

Read the full incident report →

Started May 14, 2026, 10:57 AM UTC· resolved May 14, 2026, 11:13 AM UTC

1. investigating May 14, 2026, 10:57 AM UTC

   We are currently investigating this issue.

2. identified May 14, 2026, 11:00 AM UTC

   We have identified a potential issue causing the service access problem and are working hard to address it. Please continue to monitor this page for updates.

3. resolved May 14, 2026, 11:13 AM UTC

   We can confirm normal operation, Slowness issue is resolved. We will publish the RCA

4. postmortem May 22, 2026, 08:37 PM UTC

   ## Summary Between May 17 and May 19, 2026, customers in the APAC region experienced intermittent unresponsiveness with the Harness platform. The incident occurred twice within a two-day window. In both cases, the GraphQL service became degraded, causing requests to fail or time out for affected users. The service was restored each time by restarting the impacted pods. A permanent alerting fix is in progress and expected to reach production shortly. ‌ ## Impact * APAC-region customers experienced intermittent inability to access the Harness platform UI and API * No data loss was reported; the impact was limited to service availability ‌ ## Root Cause One of the two service pods in the APAC cluster entered a degraded state where downstream calls were timing out. While the downstream services showed no visible increase in response time on their end, the pod was unable to receive responses from them in time likely due to a network-level communication issue between the pod and its downstream dependencies. ‌ ## Mitigation Identified and restarted problematic nodes. Recovery was confirmed by the engineering team before the incident was marked resolved. ## Next Steps **Improved Alerting and Proactive Detection** — Implemented new alerts for elevated response times to ensure any recurrence is detected and mitigated before impacting customers.

Read the full incident report →

Started May 12, 2026, 09:38 PM UTC· resolved May 12, 2026, 11:10 PM UTC

1. investigating May 12, 2026, 09:38 PM UTC

   We are currently investigating this issue.

2. identified May 12, 2026, 09:40 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 12, 2026, 10:50 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 12, 2026, 11:10 PM UTC

   This incident has been resolved.

5. postmortem May 20, 2026, 01:56 AM UTC

   ## Summary On May 12, 2026, Harness's Secure Connect service experienced a significant disruption that affected customers using Secure Connect for git connector checks and CI pipeline execution. Affected customers experienced connection timeouts and failures when attempting to use Secure Connect-enabled connectors, while connectors operating without Secure Connect continued to function normally. The disruption was caused by an automated infrastructure maintenance event in our Google Kubernetes Engine \(GKE\) environment that left a critical load balancer in an unreachable state. The issue was resolved by recreating the affected load balancer and updating the corresponding DNS record to restore connectivity. **Impact:** * Secure Connect-enabled git connector tests failing with connection timeouts * CI pipelines hanging or failing when Secure Connect was enabled * Connectors operating without Secure Connect were unaffected ## Root Cause The disruption was triggered by an automated GKE \(Google Kubernetes Engine\) control plane upgrade on the cluster hosting our internal Secure Connect FRPS \(Fast Reverse Proxy Server\) infrastructure. The upgrade executed in two sequential phases in rapid succession. Each phase restarts the GKE cloud controller manager, which is responsible for managing the lifecycle of Kubernetes LoadBalancer service IP addresses. As part of its normal reconciliation process, the controller performs a delete-then-re-insert cycle on IP address reservations when it restarts. The second phase of the upgrade began before the first reconciliation cycle had fully completed. This created a race condition where the controller deleted the IP address reservation for the internal FRPS load balancer but was interrupted before it could re-acquire it. No Harness application deployments caused or contributed to the issue. ‌ ## Mitigation The following steps were taken to restore service: 1. **FRPS pod restart** — An initial rollout restart of the FRPS deployment was performed. This restored client connectivity to the FRPS server but did not resolve the underlying IP orphaning issue. 2. **Load balancer recreation** — The internal Kubernetes service \(`frps-internal`\) was deleted and recreated, triggering GKE to provision a new load balancer with a new IP address 3. **DNS record update** — The Cloud DNS A record for was updated to point to the new load balancer IP. This restored end-to-end routing for all Secure Connect traffic. Full connectivity was confirmed via internal testing and subsequently verified by affected customers. ## Preventative Actions Harness is committed to preventing this class of incident from recurring. The following actions are being implemented:

Read the full incident report →

Started May 12, 2026, 05:33 PM UTC· resolved May 12, 2026, 05:55 PM UTC

1. investigating May 12, 2026, 05:33 PM UTC

   We are currently investigating this issue.

2. investigating May 12, 2026, 05:37 PM UTC

   We are continuing to investigate this issue.

3. identified May 12, 2026, 05:46 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 12, 2026, 05:52 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 12, 2026, 05:55 PM UTC

   This incident has been resolved.

6. postmortem May 13, 2026, 12:03 AM UTC

   ## **Summary** Between 10:07 AM–10:39 AM PST on Tuesday, May 12, 2026, customers using the prod1, prod2, and prod3 Production clusters experienced elevated latency and intermittent service degradation. During this timeframe, customers observed delegate timeouts, login failures, and pipeline execution failures ## **Root Cause** A recently introduced configuration change to a common infrastructure component caused unexpected resource pressure across nodes in the prod1, prod2, and prod3 production clusters. The peak traffic exacerbated the resource utilization and introduced elevated latency across several critical platform services. ## **Impact** 1. Customers in prod1, prod2, and prod3 experienced login and access failures for approximately 20 minutes. 2. Delegate connectivity was intermittently impacted during the incident window. 3. Pipeline executions and API requests experienced elevated failure rates and latency during the incident window. ## **Remediation** * Immediately Rolled back to the previous stable release, restoring customer pipeline functionality and alleviating node pressure. ## **Action Items** 1. Enhance perf testing to include such workloads so that we can catch issues before we hit production. 2. Increase capacity across clusters to make sure we have enough headroom to absorb the traffic surges.

Read the full incident report →

Started May 12, 2026, 09:50 AM UTC· resolved May 12, 2026, 02:35 PM UTC

1. investigating May 12, 2026, 04:02 PM UTC

   We are currently investigating this issue.

2. resolved May 12, 2026, 04:03 PM UTC

   This incident has been resolved.

3. postmortem May 20, 2026, 01:39 AM UTC

   ### Summary On May 12, 2026, Harness identified an issue affecting Kubernetes-based CI builds in the prod2 environment. The issue was caused by the unintended enablement of an internal feature flag associated with a CI performance optimization. Effect was seen for build steps using older `ci-1.16.81` CI addon version \(published 1 year ago\). Affected builds using older CI addon versions experienced the following behavior during the incident window: * Secrets referenced in Run step commands resolved as empty values, causing build failures No customer action is required. ### Impact * Affected Environment: prod2 * Affected Builds * Kubernetes CI builds executed during the incident window * Builds using CI addon versions older than `ci-1.16.81` * Customer Impact: * Secret expressions resolving as empty strings during execution ‌ ### Root Cause Harness introduced a CI optimization feature intended to improve Kubernetes pipeline execution performance. The feature depended on functionality available only in newer CI addon versions \(`ci-1.16.81` and later\). On May 12, 2026, the feature flag controlling this optimization was unintentionally enabled for customer accounts in the prod2 environment because of misconfigured checks. The intent of change was to verify in certain internal accounts as part of a phased roll out of this feature. ‌ For environments running older addon versions: ‌ * Runtime secret resolution did not execute correctly, causing secrets referenced in Run step commands to resolve as empty values ‌ The issue was mitigated immediately by disabling the feature flag globally. ### Mitigation Actions Harness completed the following remediation steps: ‌ * Disabled the feature flag globally * Verified successful pipeline execution after rollback * Identified and deleted all affected downloadable log entries from GCP storage * Sent communications to affected customers ### Preventive Actions Harness is implementing the following safeguards to prevent recurrence: * Enforcing addon version compatibility validation before enabling feature flags that depend on addon functionality * Documenting compatibility requirements between CI manager and addon versions

Read the full incident report →

Started May 11, 2026, 10:42 PM UTC· resolved May 11, 2026, 11:15 PM UTC

1. investigating May 11, 2026, 10:42 PM UTC

   We are currently investigating this issue.

2. monitoring May 11, 2026, 10:42 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 11, 2026, 11:15 PM UTC

   This incident has been resolved.

4. postmortem May 13, 2026, 01:53 AM UTC

   ### **Summary** On May 7, 2026, customers using the Vercel integration experienced delays in synchronizing Feature Management & Experimentation \(FME\) changes to Vercel. The issue was caused by a dependency update in the integration service that prevented outbound synchronization requests from being authenticated correctly. The issue was resolved on May 11, 2026 through a service fix and validation of synchronization workflows. ### **Impact** * Customers using the Vercel integration experienced delays propagating FME changes to Vercel * Customers observed stale configuration data in Vercel after updating changes in FME * Other FME functionality remained operational * No data loss occurred ### **Root Cause** A deployment to the `integrations-service-v2` service introduced dependency updates that removed a required AWS SDK component used during request signing and authentication for synchronization calls. As a result, outbound requests from the FME integration service to the synchronization endpoint could not be authenticated correctly, causing synchronization operations to fail. The issue was not immediately detected because: * Existing monitoring metrics did not have alerting configured * The failure mode did not surface as a critical application startup error * Integration-level automated test coverage for this synchronization workflow was insufficient ### **Resolution** The missing dependency was restored and the synchronization workflow was validated across affected integrations. Monitoring and validation checks confirmed successful synchronization behavior after deployment. ### **Preventive Actions** To prevent such issues from happening again, We are implementing the following improvements: * Enhance end-to-end integration tests for the Vercel synchronization workflow * Enhance our proactive alerts for synchronization failures and elevated error rates

Read the full incident report →

Started May 11, 2026, 09:50 AM UTC· resolved May 11, 2026, 04:59 PM UTC

1. investigating May 11, 2026, 09:50 AM UTC

   We are currently investigating this issue.

2. identified May 11, 2026, 09:51 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 11, 2026, 04:58 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 11, 2026, 04:59 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 11, 2026, 06:48 AM UTC· resolved May 11, 2026, 07:16 AM UTC

1. investigating May 11, 2026, 06:48 AM UTC

   We are currently investigating this issue.

2. identified May 11, 2026, 06:57 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 11, 2026, 07:12 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 11, 2026, 07:16 AM UTC

   This incident has been resolved.

5. postmortem Jun 23, 2026, 11:51 PM UTC

   .

Read the full incident report →

Started May 08, 2026, 06:10 PM UTC· resolved May 08, 2026, 06:10 PM UTC

1. resolved May 19, 2026, 12:33 AM UTC

   On May 8, 2026, a CI performance optimization feature flag was inadvertantly impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions

2. postmortem May 19, 2026, 12:33 AM UTC

   On May 8, 2026, a CI performance optimization feature flag was inadvertently impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions. Customers running delegate versions that did not support the new runtime secret resolution capability experienced pipeline execution failures where secrets referenced in Run step commands resolved as empty strings. The issue primarily impacted customers running older delegate versions outside the currently supported delegate release window. Customers on supported delegate versions were not impacted. The issue was mitigated by disabling the feature flag, which immediately restored the prior secret resolution behavior. No customer configuration changes were required. **Impact** * Pipeline execution failures for affected Kubernetes CI builds * Secrets referenced in Run step commands resolved as empty strings * Customers running supported delegate versions were not affected **Root Cause** A feature flag intended for a limited internal rollout was unintentionally enabled for a broader set of accounts during a routine rollout process. The feature relied on delegate-side functionality available only in newer delegate versions. Older delegate versions did not support the required runtime secret resolution behavior, resulting in pipeline failures. ‌ **Resolution** The feature flag was disabled globally, restoring the previous behavior where secrets are resolved during pipeline initialization. Impacted pipelines recovered immediately after rollback. **Preventive Actions** Harness is implementing the following improvements: * Enhancements to the feature flag approval workflow to more clearly display rollout scope. * Improved process and documentation for feature flags that depend on minimum delegate versions * Expanded monitoring and alerting for secret resolution failures in CI pipelines

Read the full incident report →

Started May 08, 2026, 05:05 PM UTC· resolved May 08, 2026, 06:53 PM UTC

1. investigating May 08, 2026, 05:05 PM UTC

   We are currently investigating this issue.

2. monitoring May 08, 2026, 06:01 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 08, 2026, 06:53 PM UTC

   This incident has been resolved.

4. postmortem May 18, 2026, 05:26 PM UTC

   ### Summary On May 8, 2026, between approximately 17:00 and 18:37 UTC, a small fraction of SDK requests to [sdk.split.io](http://sdk.split.io/) returned errors or experienced elevated latency. The cache that fronts [sdk.split.io](http://sdk.split.io/) continued to serve the vast majority of requests normally throughout the event. The issue was mitigated by increasing backend capacity, and all systems returned to baseline by 18:37 UTC. ### Root Cause [sdk.split.io](http://sdk.split.io/) is served by a CDN that caches flag data and forwards a small percentage of requests to a backend service when cached data is not available. During this event, an elevated rate of cache updates caused the CDN to forward roughly 40x its normal volume of requests to the backend. Sustained over several hours, this exceeded the backend's available capacity, leading to elevated latency, increased error rates on requests forwarded to the backend, and intermittent service restarts during recovery. ### Impact * The CDN continued to serve 97.5% of requests directly from cache as normal across the affected window. * The remaining requests forwarded to the backend saw elevated errors \(approximately 0.35% of total SDK traffic during the window\) or latency until backend capacity was restored. * SDKs continued to evaluate flags normally using their locally cached flag data; no flag evaluation correctness issues occurred. New SDK instances initializing and getting an error retried and/or signaled the timeout depending on configurations. * No data loss occurred. ### Remediation * Increased backend capacity across multiple dimensions, including service replicas, compute and memory per replica, and database connection capacity. * Service stabilized at the increased capacity profile after the changes were applied. ### Action Items * Allow additional backend capacity parameters to be tuned at runtime for quicker scaling response. * Add monitoring and alerting on identified leading indicators for earlier detection of saturation trends. * Adopt more responsive autoscaling for the backend so it can react to traffic surges more aggressively than the current autoscaler. * Improve load-shedding so the backend drops excess requests cleanly under saturation rather than queueing to failure.

Read the full incident report →

Started May 07, 2026, 08:00 AM UTC· resolved May 07, 2026, 12:54 PM UTC

1. investigating May 07, 2026, 09:08 AM UTC

   We are currently investigating this issue.

2. investigating May 07, 2026, 10:19 AM UTC

   We are continuing to investigate this issue.

3. identified May 07, 2026, 11:20 AM UTC

   We have identified the root cause as elevated queue depth and throughput saturation on the MongoDB layer, leading to processing latency. A fix is being implemented to optimize handling and restore database stability

4. identified May 07, 2026, 12:06 PM UTC

   Executions are running fine now. However, there is still some slowness in the execution graph generation so we might see some delay in the execution updates to show up in UI. But executions would work without any delay/slowness.

5. monitoring May 07, 2026, 12:09 PM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring May 07, 2026, 12:12 PM UTC

   A fix has been implemented and we are monitoring the system.

7. monitoring May 07, 2026, 12:12 PM UTC

   We are continuing to monitor for any further issues.

8. resolved May 07, 2026, 12:54 PM UTC

   This incident has been resolved.

9. postmortem May 20, 2026, 12:47 AM UTC

   Dupe of [Deployment Degradation – Failures in Prod1,2,3](https://manage.statuspage.io/pages/zmnt6tkys0q0/incidents/v4nxhkv1mvdg)

Read the full incident report →

Started May 07, 2026, 07:54 AM UTC· resolved May 07, 2026, 11:23 AM UTC

1. investigating May 07, 2026, 07:54 AM UTC

   CDS Deployment Failing in looping startegy

2. investigating May 07, 2026, 08:17 AM UTC

   We are continuing to investigate this issue.

3. investigating May 07, 2026, 08:43 AM UTC

   We are continuing to investigate this issue.

4. monitoring May 07, 2026, 09:05 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 07, 2026, 11:23 AM UTC

   This incident has been resolved.

6. postmortem May 20, 2026, 12:43 AM UTC

   ### Incident Summary On May 6 at 11:50 PM PST, we deployed a configuration change to one of our core pipeline services. This change introduced an unintended interaction with our database layer, causing a significant increase in write load. The resulting pressure degraded query and command throughput across the platform ### Root Cause The configuration change introduced a blocking condition on expression evaluation in the pipeline service. When executions encountered blocked expressions, they failed and retried repeatedly, generating a write storm against the database and the throughput went 4x ‌ ### Remediation ● Rolled back the configuration change. ● Applied database-level tuning to reduce write pressure and accelerate backlog drainage ● Performed a controlled failover to a healthy database node to restore throughput ● Scaled up database nodes to provide sufficient capacity for full recovery ‌ ### Preventive Actions To prevent from such Issues happening again, we are focussing on: 1\. Increased database resilience: We are implementing automated load-shedding thresholds that trigger on leading indicators \(replication lag, session depth, op latency\) before the database reaches saturation, preventing retry storms from compounding into full degradation events. 2\. We are optimizing databases so that we can increase write throughput by an order of magnitude and enable independent scaling of customer data workloads. This would have allowed us to drain the message backlog nearly instantaneously during this incident.

Read the full incident report →

Started May 06, 2026, 04:12 PM UTC· resolved May 06, 2026, 04:25 PM UTC

1. investigating May 06, 2026, 04:12 PM UTC

   We are currently investigating an issue where the AI Summary feature in the Harness SEI module is not available across all Insights. Our engineering team is actively working to restore the functionality as quickly as possible. We will share further updates as more information becomes available.

2. resolved May 06, 2026, 04:25 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 05, 2026, 05:14 PM UTC· resolved May 05, 2026, 05:51 PM UTC

1. investigating May 05, 2026, 05:14 PM UTC

   We are currently investigating this issue.

2. resolved May 05, 2026, 05:51 PM UTC

   This incident has been resolved.

3. postmortem May 20, 2026, 12:34 AM UTC

   ## **Summary** On May 5, 2026, customers using Infracost evaluations in the Prod2 environment experienced failures during infrastructure cost estimation workflows. Infrastructure deployments themselves continued to function normally, but pricing evaluations failed due to requests being routed to an incorrect endpoint configuration. The issue was caused by a ingress routing configuration issue during a Prod2-specific rollout. Requests to the Infracost service were routed to a default HTML login surface instead of the expected API endpoint, causing evaluation failures. ## **Impact** * Infracost evaluations failed for customers in the Prod2 environment. * Infrastructure deployments and pipeline executions continued functioning normally. * The issue was isolated to pricing and evaluation functionality only. * Prod1 and other environments were not impacted. ## **Root Cause** The incident was caused by a configuration mismatch between the Infracost plugin endpoint and the ingress routing configuration in Prod2. This resulted in invalid responses and caused Infracost evaluations to fail. ## **Remediation** The issue was mitigated by rerouting Prod2 traffic to the stable Prod1 Infracost API endpoint while validating routing behavior. Once traffic was repointed, impacted pipelines successfully completed and Infracost evaluations were restored. ## **Preventive Actions** * Add automated validation to ensure ingress routes and endpoint configurations remain aligned. * Introduce synthetic health checks for Infracost evaluation APIs across environments. * Strengthen rollout validation processes for environment-specific routing overrides. * Improve deployment guardrails to detect configuration drift between application and ingress configurations before rollout completion.

Read the full incident report →

Started May 05, 2026, 07:57 AM UTC· resolved May 05, 2026, 08:17 AM UTC

1. investigating May 05, 2026, 07:57 AM UTC

   We are currently experiencing an issue, K8s customer billing data will be stale.

2. resolved May 05, 2026, 08:17 AM UTC

   This incident has been resolved.

3. postmortem May 20, 2026, 12:02 AM UTC

   ## **Summary** On May 4, 2026, after a service deployment in Prod2, background schema migrations did not complete successfully. As a result, some cluster and perspective-related data appeared missing or stale for Elevance and a small number of other Prod2 customers. The service itself remained available, but the database schema was not updated to match the new application code. This caused downstream billing and cluster data processing jobs to fail or skip expected data updates. ## **Impact** * Affected customers saw missing or stale cluster and perspective data in the UI. * Data appeared to stop updating around April 30 for impacted accounts. * Elevance was the primary impacted customer, along with a few other Prod2 customers. * There was no full service downtime. * Once remediated, the missing data was backfilled. ## **Root Cause** During the Prod2 deployment , the background schema migration process attempted to acquire a Redis distributed lock before running Timescale database migrations. The lock acquisition failed immediately due to a `PersistentLockException`, likely because another replica or overlapping deployment process was holding the lock. Since the migration used a zero-wait lock acquisition path, it did not retry and the migration did not run. The failure was logged only as a generic warning rather than a clear production error. Because of this, the migration failure was not immediately surfaced through alerts, and the service continued running with the database schema behind the application code expectations. ## **Remediation** The service was redeployed in Prod2. On restart, the background migrations completed successfully. The team then reran the required job to backfill the missing data and restore expected cluster and perspective data visibility. ## **Preventive Actions** * Update background migration locking behavior to wait/retry when acquiring the migration lock. * Improve logging from generic warnings to explicit error logs when schema migrations are skipped. * Add alerting for failed or skipped production schema migrations.

Read the full incident report →

Started May 01, 2026, 04:28 PM UTC· resolved May 01, 2026, 08:10 PM UTC

1. investigating May 01, 2026, 04:28 PM UTC

   We are currently investigating this issue.

2. monitoring May 01, 2026, 04:56 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring May 01, 2026, 05:14 PM UTC

   We are continuing to monitor for any further issues.

4. monitoring May 01, 2026, 05:15 PM UTC

   We are continuing to monitor for any further issues.

5. monitoring May 01, 2026, 07:58 PM UTC

   We are largely mitigated and most pipelines are running normally. We are monitoring all parameters to make sure there are no issues before closing it.

6. resolved May 01, 2026, 08:10 PM UTC

   This incident has been resolved.

7. postmortem May 11, 2026, 09:08 PM UTC

   ### **Summary** A rollout involving OpenTelemetry instrumentation changes introduced a memory leak in the OTEL eBPF collector running in production clusters. Under sustained production traffic, the leak caused increasing JVM heap utilization, elevated garbage collection pressure, and eventual out-of-memory \(OOM\) conditions across several core platform services. ### **Impact** * Elevated latency and intermittent instability in Prod1, Prod2, and Prod3 * Some customers experienced slow pipeline execution and degraded responsiveness No customer data loss occurred. ### **Root Cause** The root cause was an upstream defect in the OpenTelemetry eBPF instrumentation library that introduced a memory leak under production-scale workloads. The leak continuously increased telemetry-related memory consumption, leading to sustained JVM garbage collection pressure and eventual heap exhaustion. ### **Mitigation and Recovery** **Immediate Actions** * scaled up clusters to stabilize impacted clusters * Disabled OTEL instrumentation components and restarted affected services ### **Next Steps** To prevent such issues from happening again, we are: * Enhance our load testing process to test in higher workloads to identify such issues prior to going production. * Add additional granular instrumentation to catch such issues sooner.

Read the full incident report →

Started May 01, 2026, 03:02 PM UTC· resolved May 01, 2026, 08:10 PM UTC

1. investigating May 01, 2026, 03:02 PM UTC

   We are currently investigating this issue.

2. monitoring May 01, 2026, 03:37 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring May 01, 2026, 07:58 PM UTC

   We are largely mitigated and most pipelines are running normally. We are monitoring all parameters to make sure there are no issues before closing it.

4. resolved May 01, 2026, 08:10 PM UTC

   This incident has been resolved.

5. postmortem May 11, 2026, 08:58 PM UTC

   ### **Summary** A rollout involving OpenTelemetry instrumentation changes introduced a memory leak in the OTEL eBPF collector running in production clusters. Under sustained production traffic, the leak caused increasing JVM heap utilization, elevated garbage collection pressure, and eventual out-of-memory \(OOM\) conditions across several core platform services. ### **Impact** * Elevated latency and intermittent instability in Prod1, Prod2, and Prod3 * Some customers experienced slow pipeline execution and degraded responsiveness No customer data loss occurred. ### **Root Cause** The root cause was an upstream defect in the OpenTelemetry eBPF instrumentation library that introduced a memory leak under production-scale workloads. The leak continuously increased telemetry-related memory consumption, leading to sustained JVM garbage collection pressure and eventual heap exhaustion. ### **Mitigation and Recovery** **Immediate Actions** * scaled up clusters to stabilize impacted clusters * Disabled OTEL instrumentation components and restarted affected services ### **Next Steps** To prevent such issues from happening again, we are: * Enhance our load testing process to test in higher workloads to identify such issues prior to going production. * Add additional granular instrumentation to catch such issues sooner.

Read the full incident report →

Started Apr 30, 2026, 11:09 PM UTC· resolved Apr 30, 2026, 11:43 PM UTC

1. investigating Apr 30, 2026, 11:09 PM UTC

   We are currently investigating this issue.

2. identified Apr 30, 2026, 11:19 PM UTC

   Issue has been identified and fix is underway

3. monitoring Apr 30, 2026, 11:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 30, 2026, 11:43 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 30, 2026, 04:25 PM UTC· resolved Apr 30, 2026, 05:50 PM UTC

1. investigating Apr 30, 2026, 04:25 PM UTC

   We are currently investigating this issue.

2. identified Apr 30, 2026, 05:27 PM UTC

   Issue has been identified and mitigated

3. resolved Apr 30, 2026, 07:08 PM UTC

   This incident has been resolved.

4. postmortem May 11, 2026, 08:47 PM UTC

   # **Summary** On April 30, 2026, between approximately 15:29 UTC and 17:00 UTC, customers in Prod3 experienced degradation impacting delegate connectivity, instance synchronization, pipeline executions, and connector operations due to spike in load on one of our services. . Service stability was restored through service scaling, infrastructure capacity increases, and database resource expansion. # **Impact** Customer Impact: * Delegates disconnected intermittently during the incident window * Instance synchronization operations were delayed * Some pipeline executions and connector operations experienced failures or delays Duration: * Delegate connectivity impact: ~15 minutes * Elevated service degradation: ~90 minutes # **Root Cause** The incident was caused by spike causing thread exhaustion and elevated request contention between internal services during a period of increased synchronization and delegate activity. ‌ # **Mitigation and Recovery** The following actions were taken to restore service stability: * Scaled management service replicas horizontally * Increased autoscaling thresholds and maximum replica counts * Expanded Database compute capacity * Upgraded MongoDB infrastructure components * Stabilized delegate reassignment and reconnection processing Services recovered progressively beginning at approximately 15:47 UTC, with full stability restored by ~17:00 UTC. # **Preventive Actions** To prevent such issues from happening again, We are implementing the following improvements * Improving our circuit breakers and fail-fast protections between dependent services * Enhancing monitoring and alerting for thread pool saturation and queue buildup * Increasing baseline service headroom and resiliency protections

Read the full incident report →

Started Apr 30, 2026, 03:57 PM UTC· resolved Apr 30, 2026, 04:14 PM UTC

1. investigating Apr 30, 2026, 03:57 PM UTC

   We are currently investigating this issue.

2. resolved Apr 30, 2026, 04:14 PM UTC

   This incident has been resolved.

3. postmortem May 08, 2026, 05:21 PM UTC

   ### Summary On April 30, 2026, customers with the Jira Cloud integration configured were unable to create feature flags via the UI. The issue lasted approximately 3 hours and was resolved through a rollback followed by a permanent fix deployment the same day. ### Impact * Feature flag creation in the UI failed for customers with the Jira Cloud integration installed and configured * Flag creation via API was not affected * Customers without the Jira integration were not affected * Existing feature flags, SDK evaluations, and all other platform functionality remained fully operational * No data loss occurred ### Timeline | **Time \(UTC\)** | **Event** | | --- | --- | | April 30, 13:07 | Feature flag creation began failing for affected accounts | | April 30, 15:31 | Service rolled back, restoring functionality | | April 30, 17:22 | _Identified root cause_ | | April 30, 18:59 | Permanent fix deployed and validated | ### Root Cause A service update modified how user identity is resolved when retrieving data from configured integrations. During flag creation in the UI, the system calls out to the Jira integration to power the issue-linking feature on the creation page. Under the new logic, the identity context was unavailable for this outbound call, causing the flag creation flow to fail for accounts with the Jira Cloud integration configured. The issue was limited to the UI creation path and did not affect API-based flag creation or any other platform operations. ### Remediation * Immediate: Rolled back to the previous version, restoring feature flag creation for all affected customers. * Permanent: Deployed a fix that correctly resolves user identity when retrieving data from configured integrations. ### Action Items * Improve resilience of integration paths so that failures when fetching data from connected integrations do not block core platform operations such as flag creation. * Expand automated test coverage for end-to-end scenarios involving configured integrations to prevent similar regressions.

Read the full incident report →

Started Apr 28, 2026, 06:27 PM UTC· resolved Apr 28, 2026, 06:27 PM UTC

1. resolved Apr 28, 2026, 09:48 PM UTC

   Requests made by SDKs with Rule Based Segments support could have gotten a null instead of an empty for responses with no Rule Based Segments, leading to an null pointer exception. Impact observed: 18:27 until 19:05 UTC. The impact for SDKs would have been a failure to initialize or process an update. Any change made to feature flags or RBS in an affected environment would re-generate any specific caches remaining.

2. postmortem May 08, 2026, 07:19 PM UTC

   ## Summary On April 28, 2026, a change was pushed to the Harness FME SDK API service modifying the response when there are no active Rule-based Segments to return on the SDK. The expected behavior was to return an empty list in this scenario, but a null value was served to SDKs instead, causing an error parsing the response. ## Impact * This issue impacted FME SDKs released May 2025 or after, as these support Rule-based segments. Prior versions were unaffected. * SDK instances were unable to initialize. Even if flags could be parsed, rule-based segments were not. Rendering the SDK unable to mark itself as ready and enable proper evaluations. * Client side SDKs WITH an existing cache were not impacted. The issue was specific to initializations. ## Timeline | Time \(UTC\) | Event | | --- | --- | | April 28, 18:27 | Deployed the defective SDK api version to production | | April 28, 18:35 | FME team detected the issue validating the production deployment | | April 28, 18:40 | Root cause was identified | | April 28, 18:50 | Fixed version validated e2e in lower, merged | | April 28, 19:05 | Service deployed to production successfully | | April 28, 19:08 | Fix validated in prod. | ## Root Cause When the new filter was added to filter non-active rule-based segments on initialization requests \(`rbSince=-1` on request params\), the Collect function of Golang iterables was used which returns `nil` when there are empty results. Prior to this, we just exposed the query result, which was never nil. This caused the unintended type change, leading to the issue on the SDKs related parser. ## Remediation **Fix deployed:** The team fixed the code on the service to ensure that empty lists were properly serialized as a list with no items \(`[]`\) in the responses served to SDKs. We confirmed the fix in staging and deployed to production as soon as possible. ## Action Items * Testing coverage analysis and improvements for Rule-based segments: Identify any remaining edge-cases that could be introduced by mistake and ensure we have the right coverage for all of these. * Improve continuous E2E suites visibility of the SDKs behavior: Our test applications running SDKs in them remained unaffected for the most part, given these are SDKs running emulating continuous real usage. This issue affected only initialization.

Read the full incident report →

Started Apr 28, 2026, 05:21 PM UTC· resolved Apr 28, 2026, 05:21 PM UTC

1. resolved May 02, 2026, 12:01 AM UTC

   Between April 28 and May 1, 2026, a subset of scheduled change requests failed to execute at their scheduled times. The issue was introduced by a production deployment on April 28 and fixed with a new deployment on May 1. All affected scheduled jobs that had not been manually withdrawn by customers have since been executed successfully.

2. postmortem May 06, 2026, 03:58 AM UTC

   ## Summary Between April 28 and May 1, 2026, a subset of scheduled change requests failed to execute at their scheduled times. The issue was introduced by a service update on April 28 and resolved on May 1. All affected scheduled jobs that had not been manually withdrawn by customers have since been executed successfully. ## Root Cause A service update exposed a latent bug in the path that retrieves applicable governance policies for a change request. This caused scheduled change request execution to fail, leaving affected change requests stuck in their scheduled state with no automatic retry. ## Impact * Scheduled change request execution failed silently for a subset of flags on affected accounts * Affected change requests were not applied at their scheduled time; they remained visible as scheduled in the UI * SDKs and runtime flag evaluation were not impacted * No data loss occurred; all backlogged jobs have since been processed ## Timeline | Time \(UTC\) | Event | | --- | --- | | Apr 28, 18:21 | Production deployment introduced the regression | | Apr 28, 20:31 | First scheduled change request execution fails | | May 1, 08:35 | First customer report received | | May 1, 09:43 | Escalated to engineering | | May 1, 19:43 | Fix deployed to production | | May 1, 22:28 | All stuck scheduled jobs confirmed published | ## Remediation * Deployed a forward fix addressing the underlying latent bug and adding additional protections * Reset scheduler tasks, allowing the scheduler to re-execute them; all have since been confirmed published ## Action Items * Enhance test coverage for scheduled change request execution edge cases * Add alerting for scheduled changes that have not executed within a threshold of their scheduled time, enabling proactive detection ahead of customer reports

Read the full incident report →