Harness Outage History

Started Jan 20, 2026, 11:00 AM UTC· resolved Jan 20, 2026, 02:24 PM UTC

1. investigating Jan 20, 2026, 11:00 AM UTC

   We are currently investigating an issue affecting log visibility for a few CI steps after execution completion. Our team is actively working on identifying the root cause.

2. identified Jan 20, 2026, 02:21 PM UTC

   The issue was identified in the latest log-service deployment; as a mitigation, the deployment was rolled back.

3. monitoring Jan 20, 2026, 02:21 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 20, 2026, 02:24 PM UTC

   This incident has been resolved.

5. postmortem Jan 28, 2026, 07:33 PM UTC

   ## **Summary** On January 19, 2026, some customers experienced missing logs for certain CI pipeline steps with very short execution times. The issue was related to increased latency in the log streaming process, which caused log data for affected steps to be unavailable after execution completed. The issue was identified and mitigated promptly. ## **Impact** During the incident window: * CI pipeline steps with very short execution durations may have shown incomplete or missing logs after completion. * The issue primarily affected steps that completed execution before log streaming was fully established. * Pipeline execution itself was **not impacted**, and builds continued to run successfully. There was **no impact to pipeline correctness, execution results, or customer data**, aside from log visibility for the affected steps. ## **Root Cause** The issue occurred due to increased latency in the log streaming initialization process. For very short-running steps, execution completed before the log stream was fully established. As a result, the system attempted to finalize logging before the stream was ready, leading to an empty log snapshot being recorded. This behavior caused valid log data to be unavailable for a subset of short-duration steps under specific timing conditions. ## **Mitigation** As an immediate mitigation: * The recent log service change contributing to increased stream latency was rolled back. * This action restored normal log streaming behavior and prevented further log loss. Service stability and log visibility were fully restored following the mitigation. ## **Action Items** To prevent recurrence and strengthen resiliency, the following improvements are being implemented: ‌ * Add safeguards to prevent log streams from being finalized before initialization is complete. * Improve validation logic to ensure valid log data is not overwritten by empty or incomplete snapshots. * Introduce additional protections for short-duration steps to ensure consistent log availability. * Enhance monitoring around log streaming latency to enable earlier detection of similar conditions.

Read the full incident report →

Started Jan 15, 2026, 10:25 PM UTC· resolved Jan 21, 2026, 12:58 AM UTC

1. monitoring Jan 15, 2026, 10:25 PM UTC

   GCP Billing Delay: Google Cloud is experiencing billing export delays since approximately January 10, which may result in incomplete cost data for some GCP resources.

2. resolved Jan 21, 2026, 12:58 AM UTC

   This incident has been resolved.

3. postmortem Jan 21, 2026, 03:26 AM UTC

   This was a issue with GCP cloud provider. Here is Google’s response: _We are writing to follow up on the technical issue reported on January 14, 2026, regarding delayed Billing data exports to BigQuery. We are pleased to confirm that this issue was resolved on January 18, 2026.What you need to know_ _The technical issue reported on January 14, 2026, has been resolved as of January 18, 2026. All delayed cost data is now available, and exports are functioning as expected.Thank you for your patience while we worked to fix this._

Read the full incident report →

Started Jan 09, 2026, 06:15 PM UTC· resolved Jan 09, 2026, 06:30 PM UTC

1. monitoring Jan 09, 2026, 06:00 PM UTC

   A fix has been implemented and we are monitoring the results.

2. monitoring Jan 10, 2026, 12:20 AM UTC

   We are continuing to monitor for any further issues.

3. resolved Jan 10, 2026, 12:25 AM UTC

   This incident has been resolved.

4. postmortem Jan 28, 2026, 07:23 PM UTC

   ## **Summary** On January 9, 2026, some customers experienced intermittent errors and slow responses while accessing pipeline execution details and execution lists. The issue was identified promptly and mitigated by the engineering team. Service functionality was restored within a short period. **Impact** During the incident window, a subset of users may have encountered: * Intermittent failures or delays when loading pipeline execution details. * Occasional issues viewing execution history or execution lists. Pipeline executions themselves continued to run as expected. There was **no data loss**, and no long-term impact to customer environments. ## **Root Cause** The issue was caused by elevated memory usage in a subset of service instances under load. When available memory dropped below required thresholds, certain requests related to loading execution data could not be processed successfully. Because the service instances remained partially healthy, they were not immediately recycled, resulting in intermittent request failures until mitigation was applied. ## **Mitigation** We did a rolling restart of all pods. That immediately fixed the issue. As a preventive measure we also increased the pod heap ## **Action Items** To prevent recurrence and improve resiliency, the following actions are being implemented: ‌ * Increased memory allocation to affected services to better handle peak load conditions. * Improved automatic recovery behavior for services encountering unrecoverable memory conditions. * Enhanced monitoring and alerting for application-level memory usage to enable earlier detection. * Added additional safeguards to ensure degraded instances are identified and remediated more quickly.

Read the full incident report →

Started Jan 08, 2026, 07:03 PM UTC· resolved Jan 09, 2026, 12:26 AM UTC

1. monitoring Jan 08, 2026, 07:03 PM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Jan 09, 2026, 12:26 AM UTC

   This incident has been resolved.

3. postmortem Jan 16, 2026, 05:36 AM UTC

   ### **Incident Overview** On January 8, 2026, between 10:25 AM and 10:47 AM UTC, the Harness Template Service experienced authentication failures, preventing customers from loading templates. **Impact**: Customers were unable to load templates in the Harness UI. No running pipelines were affected. ### **Root Cause** During a scheduled secret rotation, one database user credential errored out during rotation. This caused the Template Service to lose database authentication, resulting in service failures. ### **Mitigation Actions** * **Immediate Response** * Re-enabled the old database user to restore Template Service authentication and functionality. * **Resolution** * Service was fully restored within 2 minutes of detection. ### **Next Steps and Improvements** * Adding a verification step to confirm the new user is active in GCP Secret Manager before disabling the old user in the database. * Enhancing the secret rotation process to prevent similar gaps in credential updates

Read the full incident report →

Started Jan 08, 2026, 04:40 PM UTC· resolved Jan 08, 2026, 04:47 PM UTC

1. investigating Jan 06, 2026, 09:18 PM UTC

   We are investigating the issue

2. investigating Jan 06, 2026, 09:26 PM UTC

   For customers who use S3 for impressions integrations we are seeing a partial data delay. The system is currently in recovery mode, and data is being processed and backfilled. No data loss is expected and it is expected to fully recover in 24 hours. For customers that use Webhooks and other third parties (amplitude, segment et al) we are testing mitigations and will come back with an ETA.

3. identified Jan 06, 2026, 10:07 PM UTC

   For customers that use Webhooks and other third parties (amplitude, segment et al) we are in recovery mode now as well.

4. identified Jan 07, 2026, 07:57 PM UTC

   Issues with Amplitude, Segment, and custom webhooks are fully resolved as of 9:15pm PT

5. monitoring Jan 08, 2026, 05:40 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jan 09, 2026, 03:57 AM UTC

   This incident has been resolved.

7. postmortem Jan 12, 2026, 07:09 PM UTC

   ## Summary * Between **Dec 28, 2025 00:04 UTC** and **Jan 8, 2026 11:48 UTC**, impressions integration data experienced delays of varying degrees. * **Amazon S3 integrations** were impacted from Dec 28 through Jan 8, with delays reaching up to 36 hours at peak. * **Amplitude, Segment, and custom webhook integrations** were impacted from Jan 2 through Jan 7, with delays reaching up to 14 hours at peak. * A small number of customers experienced data loss due to rate limiting at their destination during recovery; the vast majority of customers received all their impressions data. ## Root Cause Significant increases in impressions volume caused our integration pipelines to reach their maximum throughput capacity. The S3 integration encountered volume growth that exceeded its processing capacity, while the Amplitude, Segment, and webhook integrations faced similar throughput constraints as traffic continued to increase. ## Impact * Outbound impressions data to S3, Amplitude, Segment, and custom webhook destinations was delayed. * Customers using these integrations would have seen data arrive later than expected. ### What was not impacted? * SDK feature flag evaluations and targeting * FME flag delivery network * Events integrations * Admin API and UI access * Customer flag configuration data ## Remediation For S3 integrations, we reordered and regrouped jobs to prioritize larger integrations, allowing them more time to complete. For Amplitude, Segment, and webhook integrations, we increased throughput through configuration changes within the data pipeline. ## Action Items * **Rebuild webhook integration architecture:** We are implementing a new architecture for Amplitude, Segment, and webhook integrations that provides better isolation from noisy neighbors and higher maximum throughput. * **Improve S3 batch processing:** We are separating batch workloads to prevent a single slow job from delaying others, with prioritization now in place for larger jobs. * **Enhanced monitoring and alerting:** New alerts have been deployed for both systems to ensure engineering teams engage with delays earlier, enabling faster recovery.

Read the full incident report →

Started Jan 07, 2026, 05:09 PM UTC· resolved Jan 07, 2026, 10:35 PM UTC

1. investigating Jan 07, 2026, 05:09 PM UTC

   FME Module UI inaccessible in prod 1 cluster

2. monitoring Jan 07, 2026, 05:17 PM UTC

   Service issues have been addressed and normal operations has been resumed. We are monitoring the service to ensure normal performance continues.

3. resolved Jan 07, 2026, 10:35 PM UTC

   This incident has been resolved.

4. postmortem Jan 07, 2026, 10:37 PM UTC

   ## Summary * On Jan 7, 2026 ,starting 13:55 UTC a limited number customers could not see the FME Module in the Harness UI. Issue was fully resolved at 19:34 UTC ## Root Cause * A configuration change controlled visibility of the FME module tile in the Harness UI. The change relied on a stale list of account IDs. This caused the FME tile to be disabled for a small set of valid accounts ## Impact * Harness Prod1 was the only affected clusters only affected limited number of customers. As a result of this configuration change, the FME Module tile was not displayed in the UI * **Note**: There were no issues with SDK evaluation or the FME flag delivery network and there was no data loss incurred as a result of this disruption ### What was not impacted? * Admin API calls * Feature flag evaluations * FME flag delivery network or SDK traffic * Customer flag configuration data ## Remediation * We reverted the Configuration change and Correct account list was applied. Thereby UI access restored for all impacted customers ## Action Items * To prevent such issues from happening again, we are enhancing internal processes\(testing and validations\) to catch such issues early on.

Read the full incident report →