Harness Outage History

Started Jul 16, 2026, 06:40 AM UTC· resolved Jul 16, 2026, 07:01 AM UTC

1. investigating Jul 16, 2026, 06:59 AM UTC

   We are currently investigating this issue.

2. resolved Jul 16, 2026, 07:01 AM UTC

   This incident has been resolved.

3. postmortem Jul 22, 2026, 04:52 PM UTC

   # Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.

Read the full incident report →

Started Jul 14, 2026, 11:25 AM UTC· resolved Jul 14, 2026, 11:57 AM UTC

1. investigating Jul 14, 2026, 11:25 AM UTC

   Certain users are unable to see feature flags in prod1 and prod2

2. monitoring Jul 14, 2026, 11:56 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 14, 2026, 11:57 AM UTC

   This incident has been resolved.

4. postmortem Jul 23, 2026, 10:32 PM UTC

   ## Summary On July 14, 2026, certain Harness Feature Flag \(FF\) Classic customers on the Prod1 and Prod2 production environments were unable to view Feature Flags in the Harness platform. Affected requests returned an authorization error \(HTTP 403\), so Feature Flags were not visible for those users until access was restored. The behaviour was caused by a planned security update that began requiring an additional Feature Flags permission for related read operations. Users whose roles did not yet include that permission were correctly denied access, which appeared as a product failure. Harness temporarily rolled back the Feature Flags service change to restore access, updated the required permissions for affected users and roles, and confirmed that Feature Flag visibility returned to normal. The stronger permission checks remain in place going forward. ## Impact During the customer-reported incident window on July 14, 2026 \(status page approximately 11:25 UTC to 11:57 UTC\): * Certain Feature Flag Classic customers on **Prod1** and **Prod2** were impacted. * Affected users could not view Feature Flags in the Harness UI / API and received **403 Forbidden** responses. * Impact was limited to users and roles that did not yet have the updated Feature Flags permission required by the security change. * A public status page update was posted for Prod1 and Prod2 Feature Flags. There was **no data loss**, no change to stored feature flag configurations, and no impact to Feature Flag evaluation for applications whose SDK credentials and permissions were unaffected. Customers and users with the required permission continued to operate normally. ## Root Cause Harness deployed a planned Feature Flags authorization update that enforces the `ff_targetgroup_view` permission on target-related read paths used when viewing Feature Flags. This enforcement is intentional and remains the expected behaviour. Users and roles that had not yet been granted `ff_targetgroup_view` received 403 responses and could not see Feature Flags. From the customer’s perspective this looked like an outage; it was an authorization denial due to missing required permissions after the security update. ## Mitigation Harness completed the following mitigation steps: * Temporarily rolled back the Feature Flags service in Prod2 and Prod1 \(and aligned Prod0\) to restore access quickly while permissions were corrected. * Updated / granted the required `ff_targetgroup_view` permission for affected users and roles. * Verified that Feature Flag visibility returned to normal and closed the status page incident. These actions restored customer access. The permission enforcement introduced by the security update **remains in effect** going forward; the lasting fix is correct permission assignment, not removal of the check. ## Preventive Actions To avoid a recurrence of this issue, Harness is taking the following actions: * Keeping the stronger Feature Flags permission checks in place as the permanent security posture. * Ensuring role and permission updates for `ff_targetgroup_view` are applied with \(or before\) similar authorization changes in production.

Read the full incident report →

Started Jul 09, 2026, 08:21 PM UTC· resolved Jul 09, 2026, 09:46 PM UTC

1. investigating Jul 09, 2026, 08:21 PM UTC

   We are currently investigating this issue.

2. monitoring Jul 09, 2026, 09:10 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jul 09, 2026, 09:46 PM UTC

   This incident has been resolved.

4. postmortem Jul 21, 2026, 05:21 AM UTC

   # Summary On July 9, 2026, some customers in the Harness Prod3 cluster experienced pipeline failures over a ~2.5-hour window, despite making no changes in their pipelines. Shell Script steps failed when fetching scripts from the Harness File Store, and Kubernetes deployment steps failed when retrieving secret encryption details. # Impact 1. **Affected users**: Some customers using the Harness Prod3 cluster reported pipeline execution failures. 2. **User impact:** Customers experienced failures in Shell Script and Kubernetes deployment steps. Impacted pipelines required manual retries after the incident was resolved. 3. **Scope:** This was not a platform-wide outage. The issue was isolated to specific APIs in our internal service within the Prod3 cluster. # Root Cause The incident was caused by degraded performance in a downstream service. This led to a storm of timeout errors in our internal service, which caused these failures. # Mitigation Engineering identified the change that caused this issue in the downstream dependency service and deployed a patch to restore it. Once deployed, pipeline executions returned to normal success rates. # Preventive Measures & Next Steps 1. Added additional monitoring and alerting on the downstream service to detect performance degradation earlier. 2. Reviewing timeout and retry logic in the internal service to improve resilience against downstream latency spikes. 3. Evaluating circuit breaker patterns to prevent cascading failures from downstream dependencies.

Read the full incident report →

Started Jul 09, 2026, 09:22 AM UTC· resolved Jul 09, 2026, 09:46 AM UTC

1. investigating Jul 09, 2026, 09:22 AM UTC

   Please check dev

Read the full incident report →

Started Jul 08, 2026, 07:49 AM UTC· resolved Jul 08, 2026, 06:56 PM UTC

1. investigating Jul 08, 2026, 07:49 AM UTC

   We are currently investigating an issue in Prod3 cluster where pipeline components are loading slower than expected.

2. monitoring Jul 08, 2026, 08:22 AM UTC

   A fix has been implemented and we are monitoring the results.

3. investigating Jul 08, 2026, 02:40 PM UTC

   We are currently investigating this issue.

4. monitoring Jul 08, 2026, 04:51 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Jul 08, 2026, 06:56 PM UTC

   This incident has been resolved.

6. postmortem Jul 14, 2026, 12:14 AM UTC

   ## Summary Between July 7 and July 8, 2026, customer accounts hosted in our Prod 3 environment experienced intermittent slowness and failures when loading pipelines and resolving templates. The underlying cause was a sharp, sustained increase in internal feature-flag lookup traffic from our template-processing service to our core platform backend service. The issue presented across two separate days. On Day 1 \(July 7\), our team restored service through infrastructure level mitigations by restarting affected services and adding capacity. Because that underlying cause was still present, the same failure mode recurred on Day 2 \(July 8\). Deeper investigation on Day 2 identified the specific feature flag and code path responsible, and engineering shipped a fix so that flag check is now served from local cache. This fix was deployed across all production clusters and fully resolved the issue; it has not recurred since We are auditing all other feature flags on the platform for the same caching gap, and are implementing process and monitoring improvements described in the Preventive Measures section below to catch this class of issue earlier and reduce the risk of recurrence. ## Impact **Symptoms:** Intermittent slowness or failure loading pipelines; slow or failed template validation and resolution; intermittent connectivity/timeout errors between platform services. **Functional impact:** Pipeline executions and pipeline-studio operations that depend on template resolution were delayed or failed during the active windows described in the timeline above. Some executions required manual retry ## Root cause Our core backend service could not keep up with the request arrival rate because of a sustained, uncached feature-flag lookup driving a large and growing volume of internal traffic following a recent release. ‌ ## Mitigation * Immediate mitigation \(Day 1 and Day 2\): removed the saturated backend instance\(s\) from service and added capacity to relieve the acute bottleneck. * Permanent fix \(Day 2\): updated the template-processing service so the responsible feature-flag check reads from local cache instead of calling the backend service on every lookup, eliminating the excess internal traffic at its source. * The fix was deployed to all production clusters and verified through a return to normal internal traffic volumes and response times. ## Preventive Measures & Next Steps * Feature-flag caching audit: We are reviewing all feature flags used in high-frequency code paths across the platform to identify and close any similar caching gaps before they can cause a repeat of this issue. * Release validation: We are strengthening pre-release load testing for new feature-flag-gated code paths so that call-volume regressions of this kind are caught before reaching production. * Capacity alerting: We are adding proactive alerting on backend request-queue saturation so that this class of bottleneck is caught and mitigated automatically, before it affects customer-facing response times.

Read the full incident report →

Started Jul 07, 2026, 10:40 AM UTC· resolved Jul 07, 2026, 02:10 PM UTC

1. investigating Jul 07, 2026, 10:40 AM UTC

   We are currently investigating an issue in EU1 where users are having issues using IACM module.

2. monitoring Jul 07, 2026, 02:00 PM UTC

   We have implemented a fix on our side which has resolved this issue. We are continuing to monitor this issue.

3. resolved Jul 07, 2026, 02:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 07, 2026, 10:22 AM UTC· resolved Jul 07, 2026, 12:36 PM UTC

1. investigating Jul 07, 2026, 10:22 AM UTC

   We are currently investigating an issue in Prod3 cluster where pipeline components are loading slower than expected.

2. monitoring Jul 07, 2026, 12:19 PM UTC

   A fix has been implemented in one of our microservice responsible for dealing with pipeline templates which has resolved the issue. We are continuing to monitor the issue.

3. resolved Jul 07, 2026, 12:36 PM UTC

   This incident has been resolved.

4. postmortem Jul 14, 2026, 12:04 AM UTC

   ## Summary Between July 7 and July 8, 2026, customer accounts hosted in our Prod 3 environment experienced intermittent slowness and failures when loading pipelines and resolving templates. The underlying cause was a sharp, sustained increase in internal feature-flag lookup traffic from our template-processing service to our core platform backend service. The issue presented across two separate days. On Day 1 \(July 7\), our team restored service through infrastructure level mitigations by restarting affected services and adding capacity. Because that underlying cause was still present, the same failure mode recurred on Day 2 \(July 8\). Deeper investigation on Day 2 identified the specific feature flag and code path responsible, and engineering shipped a fix so that flag check is now served from local cache. This fix was deployed across all production clusters and fully resolved the issue; it has not recurred since We are auditing all other feature flags on the platform for the same caching gap, and are implementing process and monitoring improvements described in the Preventive Measures section below to catch this class of issue earlier and reduce the risk of recurrence. ## Impact **Symptoms:** Intermittent slowness or failure loading pipelines; slow or failed template validation and resolution; intermittent connectivity/timeout errors between platform services. **Functional impact:** Pipeline executions and pipeline-studio operations that depend on template resolution were delayed or failed during the active windows described in the timeline above. Some executions required manual retry ## Root cause Our core backend service could not keep up with the request arrival rate because of a sustained, uncached feature-flag lookup driving a large and growing volume of internal traffic following a recent release. ‌ ## Mitigation * Immediate mitigation \(Day 1 and Day 2\): removed the saturated backend instance\(s\) from service and added capacity to relieve the acute bottleneck. * Permanent fix \(Day 2\): updated the template-processing service so the responsible feature-flag check reads from local cache instead of calling the backend service on every lookup, eliminating the excess internal traffic at its source. * The fix was deployed to all production clusters and verified through a return to normal internal traffic volumes and response times. ## Preventive Measures & Next Steps * Feature-flag caching audit: We are reviewing all feature flags used in high-frequency code paths across the platform to identify and close any similar caching gaps before they can cause a repeat of this issue. * Release validation: We are strengthening pre-release load testing for new feature-flag-gated code paths so that call-volume regressions of this kind are caught before reaching production. * Capacity alerting: We are adding proactive alerting on backend request-queue saturation so that this class of bottleneck is caught and mitigated automatically, before it affects customer-facing response times.

Read the full incident report →

Started Jul 02, 2026, 03:48 PM UTC· resolved Jul 03, 2026, 04:11 PM UTC

1. monitoring Jul 02, 2026, 03:48 PM UTC

   A fix has been implemented and we are monitoring the results.

2. monitoring Jul 02, 2026, 08:10 PM UTC

   We seeing the backlogged items worked through and continuing to monitor results.

3. monitoring Jul 02, 2026, 11:31 PM UTC

   We are continuing to monitor results while the backlog is worked through.

4. resolved Jul 03, 2026, 04:11 PM UTC

   This incident has been resolved.

5. postmortem Jul 09, 2026, 07:24 PM UTC

   **Incident Summary** Harness FME \(Feature Management & Experimentation\) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline. **Root Cause** The core issue was a temporary **performance degradation on the provider's infrastructure**. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue. **Impact** * The export backlog accumulated, peaking at more than 6 hours behind the planned schedule. * No data loss was detected during the incident. ‌ **Mitigation** 1. **Adjusted Temporal workflows:** Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300. 2. **Expanded resource capacity.** 3. **Traffic redirection:** Systematically shifted export workloads over to an optimized in-house cluster. 4. **Manual intervention:** Terminated stalling workflows stuck in retry loops from failed metadata lookups. ‌ **Next Steps** Complete the migration of all remaining export jobs to the internal analytics cluster \(In progress\).

Read the full incident report →

Started Jun 30, 2026, 09:16 AM UTC· resolved Jun 30, 2026, 03:39 PM UTC

1. investigating Jun 30, 2026, 09:16 AM UTC

   Dev Statuspage incident - Testing

Read the full incident report →

Started Jun 29, 2026, 02:56 PM UTC· resolved Jun 29, 2026, 03:41 PM UTC

1. investigating Jun 29, 2026, 02:56 PM UTC

   We are currently investigating this issue.

2. identified Jun 29, 2026, 02:57 PM UTC

   Harness is investigating an issue affecting our Prod 4 environments. Our log-services are affected and harness is currently working on restoring function. This can affect our CI, Chaos, IACM modules.

3. identified Jun 29, 2026, 03:02 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jun 29, 2026, 03:20 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Jun 29, 2026, 03:20 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jun 29, 2026, 03:41 PM UTC

   This incident has been resolved.

7. postmortem Jul 02, 2026, 12:25 AM UTC

   ### **Summary** During a planned security improvement to rotate service authentication secrets in the Prod 4 environment, an incorrect secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, impacting CI initialization, delegate task creation, and log streaming for certain pipeline executions. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** Prod 4 **Customer-visible impact:** * CI builds failed during initialization with errors such as: * _“Could not fetch token from log service”_ * _“token in request not authorized for receiving tokens”_ \(HTTP 400\) * CD and IaCM pipeline stages continued to execute but did not stream logs. * Some Shell Script steps failed during execution. * Delegate tasks could not be created or assigned for some pipeline executions. No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the Prod 4 environment. During the rotation process, an incorrect secret value was inadvertently configured for one of the services. Instead of applying the newly generated secret in the expected format, a default/incorrect value was introduced. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. * Validating recovery through CI pipeline execution tests. * Verifying CD and IaCM pipeline execution and log streaming. * Executing a comprehensive validation suite to confirm platform functionality before closing the incident. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.

Read the full incident report →

Started Jun 26, 2026, 07:20 AM UTC· resolved Jun 26, 2026, 07:31 AM UTC

1. investigating Jun 26, 2026, 07:20 AM UTC

   We are currently experiencing intermittent login issues with the Traceable cluster due to an issue with our login provider. We are actively working with the provider to resolve the issue. This incident does not impact data ingestion, and all ingestion pipelines continue to function normally. We will provide updates as we have more information.

2. identified Jun 26, 2026, 07:20 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jun 26, 2026, 07:29 AM UTC

   Issue has been fixed and we are closely monitoring

4. monitoring Jun 26, 2026, 07:29 AM UTC

   We are continuing to monitor for any further issues.

5. resolved Jun 26, 2026, 07:31 AM UTC

   This incident has been resolved.

6. postmortem Jun 30, 2026, 05:13 PM UTC

   ## **Summary** ‌ _Between **06:50 UTC and 07:20 UTC on June 26, 2026**, customers experienced intermittent login failures while accessing Traceable environments across multiple US clusters. The incident was traced to an issue with the external authentication provider \(Auth0\), where elevated socket timeouts caused increased login latency and authentication request failures. Login functionality gradually recovered as the upstream issue stabilized, and the incident was resolved after successful login validation across multiple impacted clusters._ ## **Root Cause** _The root cause was an issue with the external authentication provider \(Auth0\), which experienced elevated socket timeouts while processing authentication requests. These upstream timeouts increased login latency and caused intermittent authentication failures across multiple US-region clusters. Since authentication requests depended on the external provider, affected login attempts failed despite Traceable platform services remaining healthy. The incident was resolved once the upstream authentication service recovered and login requests consistently completed successfully._ ## **Impact** _Starting at approximately 06:50 UTC, customers experienced intermittent login failures when accessing Traceable environments across multiple US clusters. The issue affected user authentication, preventing some users from accessing the platform while underlying application services remained operational. Login functionality progressively recovered during the incident, and normal authentication was restored by 07:20 UTC._ ## **Remediation** _The engineering team worked with the external authentication provider while continuously monitoring authentication health across affected clusters. Login functionality was validated through platform metrics and manual verification across representative environments. After confirming consistent authentication success across impacted clusters, the incident was declared resolved._ ## **Action Items** _To prevent such issues going forward, Harness will,_ _Increase authentication resilience: Evaluate improvements to authentication request handling, including timeout tuning, retry strategies where appropriate, and graceful degradation for transient upstream failures._

Read the full incident report →

Started Jun 24, 2026, 12:03 PM UTC· resolved Jun 24, 2026, 02:37 PM UTC

1. investigating Jun 24, 2026, 12:19 PM UTC

   Harness login is failing. We are investigating the problem.

2. monitoring Jun 24, 2026, 12:26 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Jun 24, 2026, 12:27 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Jun 24, 2026, 02:37 PM UTC

   This incident has been resolved.

5. postmortem Jul 02, 2026, 12:26 AM UTC

   ### **Summary** During a planned security improvement to rotate service authentication secrets in the EU1 environment, an out of sequence secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, causing login failures. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** EU1 **Customer-visible impact: Login Failures** No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the EU1 . During the rotation process, an secret value was inadvertently misconfigured for one of the services. There was a newer format of secret and had to be applied in a certain order. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.

Read the full incident report →

Started Jun 23, 2026, 06:46 AM UTC· resolved Jun 30, 2026, 04:33 PM UTC

1. identified Jun 23, 2026, 06:46 AM UTC

   We have identified an issue affecting a small subset of accounts in the EU1 region. As a result, data ingestion for impacted integrations may have stopped.

Read the full incident report →

Started Jun 19, 2026, 09:10 AM UTC· resolved Jun 19, 2026, 09:29 AM UTC

1. investigating Jun 19, 2026, 09:10 AM UTC

   Testing Dev in QA

Read the full incident report →

Started Jun 17, 2026, 04:44 PM UTC· resolved Jun 17, 2026, 05:50 PM UTC

1. investigating Jun 17, 2026, 04:44 PM UTC

   We are currently investigating this issue.

2. identified Jun 17, 2026, 04:45 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Jun 17, 2026, 04:45 PM UTC

   IDP Systems' communication was interrupted due to an update. Harness is rolling back changes. Impacted environments - prod0, prod1, prod2

4. monitoring Jun 17, 2026, 04:47 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Jun 17, 2026, 05:50 PM UTC

   This incident has been resolved.

6. postmortem Jun 24, 2026, 12:50 AM UTC

   # **Customer RCA: Executive Summary** On June 17, 2026, between 9:30 PM and 10:15 PM IST, customers in certain production environments experienced failures when accessing IDP \(Internal Developer Portal\) workflows. The issue was caused by a configuration mismatch in service authentication settings. ## **Impact** * **Affected Service:** IDP workflows * **Duration:** Approximately 45 minutes \(9:30 PM - 10:15 PM IST\) * **Customer Impact:** Users were unable to load IDP workflows during this period. * **Data Loss:** None * **Security Impact:** None ## **Root Cause** During a planned service deployment, a configuration mismatch occurred in the authentication settings used for internal service-to-service communication. Specifically, the authentication credentials configured in the newly deployed service did not match the credentials expected by existing services in prod environments. This mismatch caused authentication failures when services attempted to communicate with each other, resulting in workflow loading failures for customers. ## **Mitigation** The issue was resolved by reverting the configuration change to restore the previous working authentication settings. The engineering team was proactively monitoring logs immediately after deployment and identified error patterns several minutes before the first customer report. ## **Action Items/Next Steps** 1. **Enhanced Pre-Deployment Validation:** Automated validation checks will be introduced to verify that all required configurations, secrets, and service dependencies are correctly provisioned and consistent across services before any production deployment. 2. **Improved Deployment Procedures:** Updated deployment checklists with explicit verification of service authentication settings across services before rollout.

Read the full incident report →

Started Jun 17, 2026, 01:42 PM UTC· resolved Jun 17, 2026, 05:13 PM UTC

1. investigating Jun 17, 2026, 01:42 PM UTC

   We are investigating an issue affecting artifact downloads from Maven Central following recent rate-limiting changes implemented by Sonatype. Sonatype has recently implemented rate limits and builds downloading artifacts from central sonatype https://central.sonatype.org/faq/429-contact-support/. Customers downloading artifacts would be impacted. We are actively engaging with Sonatype to mitigate the issue here.

2. identified Jun 17, 2026, 02:03 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jun 17, 2026, 02:21 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Jun 17, 2026, 03:21 PM UTC

   Maven Central is rate-limiting requests originating from our us-central1 NAT egress IP. Other regions (us-west1, us-east5) are not affected. The Maven/Sonatype team has been engaged and is actively working on lifting the rate limit for our IP range.

5. monitoring Jun 17, 2026, 04:12 PM UTC

   We are continuing to monitor for any further issues.

6. resolved Jun 17, 2026, 05:13 PM UTC

   This incident has been resolved.

7. postmortem Jun 23, 2026, 11:51 PM UTC

   ## Summary On June 17, 2026, a subset of Harness Hosted CI customers experienced intermittent build failures when downloading Java/Gradle dependencies from Maven Central. Affected builds failed with `429 Too Many Requests` errors and could not proceed until the issue was resolved. The incident was not caused by a Harness code change or platform defect. It was triggered by a policy change implemented by Sonatype \(the operator of Maven Central\) that tightened IP-based rate limits for unauthenticated access. Because Harness Hosted CI routes builds through shared network egress infrastructure, the aggregate outbound traffic from multiple customers sharing a common IP address exceeded the new, lower thresholds — causing all builds behind that IP to be temporarily blocked. This was an industry-wide change. Other CI/CD platforms experienced identical incidents in the weeks prior to Harness being affected. ## Root Cause **Primary cause:** Sonatype tightened IP-based rate limiting policies on Maven Central \([repo1.maven.org](http://repo1.maven.org) and [repo.maven.apache.org](http://repo.maven.apache.org)\). Harness Hosted CI uses shared network egress infrastructure where multiple customer build environments route outbound traffic through common IP addresses. When the aggregate Maven Central request volume from all customers behind a given IP exceeded Sonatype's revised threshold, that IP was hard-blocked for approximately 30 minutes, causing all builds attempting to download Maven dependencies to fail with `429` errors. ‌ ## Mitigation **Immediate actions taken during the incident:** 1. **Regional redistribution:** For customers experiencing active failures, builds were routed to infrastructure regions whose outbound IPs had not yet hit the rate limit threshold, providing immediate relief for those customers. 2. **Sonatype engagement and IP whitelisting:** The Harness team contacted Sonatype support and provided the full list of Harness Hosted CI outbound IP addresses for whitelisting. Sonatype applied a "warning" rate-limit policy in place of hard blockingeliminating the `429` errors across all affected regions. Post-mitigation validation confirmed zero throttling across all tested IPs. ## Next Steps The following actions are in progress or planned to prevent recurrence and reduce exposure to similar incidents: **Near-term \(in progress\):** * Harness is in the process of establishing a commercial agreement with Sonatype for elevated rate limits on Maven Central and advance notification of future policy changes. * Harness is engaging other major dependency registries — including Docker Hub, npm, GitHub Packages, and the Gradle Plugin Portal — to proactively whitelist Harness egress IPs before similar incidents can occur. * We are publishing documentation on best practices for managing external dependency access in Hosted CI, including guidance on dependency proxies, mirrors, and caching strategies. * Harness plans to implement caching infrastructure for common dependencies at the platform level \(using Harness Artifact Registry\), which will significantly reduce direct Maven Central request volume and provide resilience against future external rate-limit changes. * We are enhancing our alerting alerting for `429` error patterns from external registries so that future rate-limit incidents are detected internally before customers are impacted. ‌ ## What This Means for You **To reduce future exposure**, we recommend: * **Enable Cache Intelligence** in Harness CI to cache dependency artifacts between builds, reducing how often Maven Central is contacted. * **Use Harness Save/Restore Cache steps** to persist your local dependency cache across pipeline runs. If you have questions or need help implementing any of these recommendations, please contact your Customer Success Manager or reach out to Harness Support.

Read the full incident report →

Started Jun 16, 2026, 10:30 AM UTC· resolved Jun 16, 2026, 10:54 AM UTC

1. investigating Jun 16, 2026, 10:30 AM UTC

   We are currently investigating this issue.

Read the full incident report →

Started Jun 08, 2026, 05:32 PM UTC· resolved Jun 08, 2026, 07:50 PM UTC

1. monitoring Jun 08, 2026, 05:32 PM UTC

   We are continually monitoring this issue.

2. resolved Jun 08, 2026, 07:50 PM UTC

   This incident has been resolved.

3. postmortem Jun 24, 2026, 12:40 AM UTC

   ## Summary On June 8, 2026, a subset of Harness customers experienced UI degradation in the Harness Code Repository module. Affected users saw broken styling — most notably, the Create Repository control was visually impaired and difficult to use. The issue was not a full outage; core functionality remained available Users who logged out and back in, or opened the application in an incognito/private window, were not affected — pointing to browser-stored state as the failure mechanism. ## Root Cause **Primary cause:** A change to the Harness Platform UI framework altered the default theme fallback from a valid, deployable CSS theme to a "system preference" token — a value that instructs the browser to follow the operating system's light/dark mode setting. While this token is valid as a user _preference_, it is not itself a CSS theme: no corresponding stylesheet bundle exists for it. When users without a previously saved theme preference loaded the Harness UI, this unresolvable token was immediately written to their browser's local storage and applied to the page. The result was missing CSS design tokens, which broke the visual rendering of Harness Code. Because the invalid value was persisted in browser storage, it continued affecting those users across subsequent page navigations until their session was cleared. ‌ ## Mitigation **Immediate actions:** The Harness team rolled back the Platform UI to the last known good version on all affected production environments. This stopped new users from hitting the broken default and, once users cleared their session \(via logout/login\), restored correct rendering. Affected users were advised to log out and log back in to clear the stale value from browser storage. Opening a new incognito/private window also restored correct behavior immediately. A targeted fix reverting the default theme to the correct value was merged and deployed. This permanently addressed the root cause without requiring users to take any action. ## Next Steps The following actions are in progress or planned to prevent recurrence: ‌ * **Default theme reverted:** The platform UI wrapper now uses a valid, CSS-resolvable theme as the default fallback. `system` preference tokens will no longer be used as raw defaults. \(Done\) * **Browser storage cleanup:** A cleanup was deployed to migrate any stale invalid theme values already persisted in affected users' browsers, resolving the issue without requiring manual logout. \(Done\) * **Defensive validation in the theme provider:** Adding input validation so that any theme value — whether loaded from storage or applied as a default — is checked against the set of known valid CSS themes before being written to storage or applied to the DOM. Invalid values will fall back safely to a known-good theme. * **Expanded test coverage:** Adding automated end-to-end tests that exercise the theme initialization path for users across both the new and legacy UI experiences, including first-time users with no previously saved preference. * **Client-side monitoring for theme resolution failures:** Adding telemetry to detect when an invalid theme value is applied, enabling the team to catch styling regressions before they are reported by customers.

Read the full incident report →

Started Jun 05, 2026, 10:56 PM UTC· resolved Jun 06, 2026, 12:30 AM UTC

1. investigating Jun 05, 2026, 10:56 PM UTC

   We are currently investigating this issue.

2. monitoring Jun 06, 2026, 12:01 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 06, 2026, 12:30 AM UTC

   This incident has been resolved.

4. postmortem Jun 19, 2026, 10:52 AM UTC

   ## **Summary** _Between 13:30 UTC on June 5, 2026, and 00:27:00 UTC on June 6, 2026, customers using Harness IDP experienced intermittent 503 errors when accessing IDP web pages and APIs. The incident was mitigated through a configuration update and normal operation was restored by 00:27 UTC._ ## **Root Cause** _The Root Cause was traced back to a performance optimization update deployed around 10:35 UTC on June 5, 2026. This network transport configuration change altered idle TCP connection behavior, causing long-lived connections to be terminated prematurely under certain conditions. Consequently, requests failed to reach backend services, leading to intermittent 503 errors in the Harness IDP service. Stability was restored by recalibrating connection timeouts and lifecycle settings to synchronize connection management across the network path and eliminate stale connection reuse._ ## **Impact** _Starting at approximately ~13:30 UTC, customers experienced intermittent 503 service errors while using the Harness Internal Developer Portal \(IDP\) until we resolved the issue at 00:27:00 UTC. The issue impacted both UI and API traffic, resulting in failed requests and a degradation of service availability until mitigation measures were implemented._ ## **Remediation** _The team implemented a configuration change to connection management settings, reducing idle connection timeouts and limiting connection lifetime._ _This prevented the reuse of stale connections and restored service stability._ ## **Action Items** 1. _**Enhance monitoring & alerting** – Enhance synthetic alerting and coverage for upstream connection failures, connection resets, and elevated 503 error rates to enable earlier detection of network transport issues._ 2. _**Pre-production validation of networking changes** – Expand pre-production checks for configuration alignment and testing for network timeout and connection management scenarios to reduce the risk of similar issues in the future._

Read the full incident report →

Started Jun 04, 2026, 02:33 PM UTC· resolved Jun 04, 2026, 04:38 PM UTC

1. identified Jun 04, 2026, 02:33 PM UTC

   Instability in streaming connections could cause a delay in propagating updates. All SDKs are designed to poll against the CDN on streaming disconnects and instability scenarios. This could also cause some noise in the logs as SDKs try to reconnect to streaming. But updates are unaffected.

2. monitoring Jun 04, 2026, 03:13 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 04, 2026, 04:38 PM UTC

   This incident has been resolved.

4. postmortem Jul 01, 2026, 11:19 PM UTC

   ### Summary On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects. ‌ ‌ ### Impact * **Region affected:** us-east-1 \(other regions — EU, APAC — remained healthy\) * **Customer impact:** SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation ### Root Cause The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout. ### Remediation Actions Taken 1. Service was rolled back to the previously stable version. 2. Increased capacity to handle the surge of connections. ### Preventive Measures / Next Steps To prevent such issues happening again, Harness is working on several optimizations: * **Enhance** deployments to do validation before broader rollout to catch capacity pressure early. * **Improve** connection handling during deployments or more conservative deployment pacing during high-load periods. * **Improved SDK behavior on reconnect**: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects

Read the full incident report →

Started Jun 02, 2026, 06:58 PM UTC· resolved Jun 02, 2026, 08:10 PM UTC

1. investigating Jun 02, 2026, 06:58 PM UTC

   Please ignore this incident.

2. identified Jun 02, 2026, 07:16 PM UTC

   The issue has been identified and a fix is being implemented.

Read the full incident report →

Started Jun 01, 2026, 10:01 AM UTC· resolved Jun 01, 2026, 01:47 PM UTC

1. investigating Jun 01, 2026, 10:01 AM UTC

   We are currently investigating this issue.

2. monitoring Jun 01, 2026, 11:05 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 01, 2026, 01:47 PM UTC

   This incident has been resolved.

4. postmortem Jun 24, 2026, 10:41 PM UTC

   Similar to the incident prior to this on 05/27

Read the full incident report →

Started May 27, 2026, 02:20 PM UTC· resolved May 27, 2026, 11:20 PM UTC

1. investigating May 27, 2026, 02:20 PM UTC

   We are currently investigating this issue.

2. monitoring May 27, 2026, 02:41 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 27, 2026, 11:20 PM UTC

   This incident has been resolved.

4. postmortem Jun 24, 2026, 03:03 AM UTC

   ## Summary On May 27, 2026, customers using Harness Pipelines on the legacy log-storage backend experienced intermittent failures and missing pipeline logs in the UI. **No log data was permanently lost** ## Root cause The root cause was CPU saturation of the log service's underlying cache infrastructure, triggered by an automation script from one enterprise customer that opened a large number of long-lived log-streaming connections without closing them. ## Mitigation **Immediate actions taken during the incident:** ‌ 1. **Rate limiting applied** Traffic from the automation generating the runaway connections was rate-limited at the network layer, reducing new connection creation and allowing the CPU to begin recovering. 2. **Streaming connection timeout reduced:** The maximum lifetime for legacy streaming connections was reduced from 1 hour to 10 minutes across production environments, limiting how long any single connection can remain open and reducing the steady-state connection count. 3. **Affected customer migrated to newer storage infra:** The enterprise customer most affected by the incident was migrated to the new log storage path infrastructure. Once migrated, their logs were immediately accessible, confirming that no log data had been lost. The cache CPU returned to normal levels \(~90% sustained CPU dropped\) following this migration. ## Next Steps The following actions are in progress or planned to prevent recurrence: * **Rate limiting applied** for the specific traffic pattern that triggered this incident. \(Done\) * **Streaming connection timeout reduced to 10 minutes** across primary production environments. \(Done\) * **Complete migration to new log storage infra :** The primary long-term fix is finishing the migration of all remaining accounts * **Enhance monitoring alerts:** A broader audit of all monitoring alert rules is underway to identify any other critical detection paths that may be silently disabled. * **Extend streaming connection timeout reduction globally:** The 10-minute maximum streaming connection lifetime is being rolled out to all remaining environments \(including development and free-tier clusters\) to ensure consistent protection. * **Per-account rate limiting on streaming connections:** We are adding an application-level cap on concurrent streaming connections per account. The initial implementation will be observe-only \(logging warnings when thresholds are approached\) to gather data on real-world usage before converting to hard enforcement. * **Improved error messaging:** When log operations fail due to underlying infrastructure issues \(cache timeouts, I/O errors\), the error surfaces to users and logs will be updated to accurately reflect the infrastructure cause, rather than showing a generic "stream not found" message that obscures the true reason for failure.

Read the full incident report →

Started May 26, 2026, 03:43 PM UTC· resolved May 26, 2026, 03:58 PM UTC

1. investigating May 26, 2026, 03:43 PM UTC

   Testing Dev Status Page incident message

Read the full incident report →

Started May 21, 2026, 01:55 PM UTC· resolved May 21, 2026, 04:34 PM UTC

1. investigating May 21, 2026, 01:55 PM UTC

   We are currently investigating an slowness issue in pipeline execution ui.

2. investigating May 21, 2026, 02:15 PM UTC

   Our investigation has revealed that the pipeline execution are happening as expected in the backend, The slowness is limited to updates in pipeline UI only.

3. identified May 21, 2026, 02:33 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 21, 2026, 02:49 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 21, 2026, 04:34 PM UTC

   This incident has been resolved.

6. postmortem Jun 01, 2026, 08:37 AM UTC

   # **Summary** On May 20 and May 21, 2026, customers on the Prod3 cluster experienced degraded pipeline performance on two separate occasions. In both cases, pipeline executions ran slower than usual and execution graph rendering was delayed because database operations were being queued on the backing MongoDB instance. Both incidents were mitigated by adjusting database write settings and adding capacity headroom, which relieved pressure on MongoDB and allowed normal operations to resume. # **Root Cause** A higher-than-usual rate of concurrent updates against a frequently modified document used during pipeline execution produced a high rate of write conflicts in MongoDB. Each conflict triggered an automatic retry, which compounded under load and drove CPU utilization on the database to a sustained high level. As CPU saturated: * MongoDB started queuing incoming commands instead of executing them immediately. * Operations dependent on those writes — pipeline step transitions and execution graph generation — slowed down or stalled while waiting for the queue to drain. # **Customer Impact** Customers on Prod3 were impacted during two windows: * May 20, 2026: ~3:15 AM to ~4:45 AM PST \(~1h 30m\). * May 21, 2026: ~6:42 AM to ~7:47 AM PST \(~1h 5m\). In both windows: * Pipeline executions ran significantly slower, with delays in step initialization and execution graph generation. * Pipelines with tight configured timeouts may have timed out as a result of the slowness. No execution failures were observed beyond timeout-related effects, and no full outage occurred. All Harness modules that rely on Pipelines on Prod3 were affected for the duration of each incident. # **Resolution** After identifying that database contention was the bottleneck, we adjusted the pipeline service's database write settings to reduce overhead on each operation and added additional database capacity to absorb the load. Together these reduced write conflicts and allowed the queued commands to drain, after which pipeline operations returned to normal. # **Prevention and Improvements** 1. Strengthen concurrency control around the high-contention update path so that concurrent updates to the same document no longer race in a way that produces write conflicts at scale. 2. Strengthen load and concurrency testing on hot execution paths so similar contention patterns are caught before they affect production traffic. 3. Improve proactive alerting on database write conflicts, queued commands, and sustained CPU utilization so the team can intervene before customer-facing latency is affected.

Read the full incident report →