Secret resolution failures in CI executions - PROD2

Update timeline

1. investigating May 12, 2026, 04:02 PM UTC

   We are currently investigating this issue.

2. resolved May 12, 2026, 04:03 PM UTC

   This incident has been resolved.

3. postmortem May 20, 2026, 01:39 AM UTC

   ### Summary On May 12, 2026, Harness identified an issue affecting Kubernetes-based CI builds in the prod2 environment. The issue was caused by the unintended enablement of an internal feature flag associated with a CI performance optimization. Effect was seen for build steps using older `ci-1.16.81` CI addon version \(published 1 year ago\). Affected builds using older CI addon versions experienced the following behavior during the incident window: * Secrets referenced in Run step commands resolved as empty values, causing build failures No customer action is required. ### Impact * Affected Environment: prod2 * Affected Builds * Kubernetes CI builds executed during the incident window * Builds using CI addon versions older than `ci-1.16.81` * Customer Impact: * Secret expressions resolving as empty strings during execution ‌ ### Root Cause Harness introduced a CI optimization feature intended to improve Kubernetes pipeline execution performance. The feature depended on functionality available only in newer CI addon versions \(`ci-1.16.81` and later\). On May 12, 2026, the feature flag controlling this optimization was unintentionally enabled for customer accounts in the prod2 environment because of misconfigured checks. The intent of change was to verify in certain internal accounts as part of a phased roll out of this feature. ‌ For environments running older addon versions: ‌ * Runtime secret resolution did not execute correctly, causing secrets referenced in Run step commands to resolve as empty values ‌ The issue was mitigated immediately by disabling the feature flag globally. ### Mitigation Actions Harness completed the following remediation steps: ‌ * Disabled the feature flag globally * Verified successful pipeline execution after rollback * Identified and deleted all affected downloadable log entries from GCP storage * Sent communications to affected customers ### Preventive Actions Harness is implementing the following safeguards to prevent recurrence: * Enforcing addon version compatibility validation before enabling feature flags that depend on addon functionality * Documenting compatibility requirements between CI manager and addon versions