Harness experienced a minor incident on May 19, 2026, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved May 19, 2026, 12:33 AM UTC
On May 8, 2026, a CI performance optimization feature flag was inadvertantly impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions
- postmortem May 19, 2026, 12:33 AM UTC
On May 8, 2026, a CI performance optimization feature flag was inadvertently impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions. Customers running delegate versions that did not support the new runtime secret resolution capability experienced pipeline execution failures where secrets referenced in Run step commands resolved as empty strings. The issue primarily impacted customers running older delegate versions outside the currently supported delegate release window. Customers on supported delegate versions were not impacted. The issue was mitigated by disabling the feature flag, which immediately restored the prior secret resolution behavior. No customer configuration changes were required. **Impact** * Pipeline execution failures for affected Kubernetes CI builds * Secrets referenced in Run step commands resolved as empty strings * Customers running supported delegate versions were not affected **Root Cause** A feature flag intended for a limited internal rollout was unintentionally enabled for a broader set of accounts during a routine rollout process. The feature relied on delegate-side functionality available only in newer delegate versions. Older delegate versions did not support the required runtime secret resolution behavior, resulting in pipeline failures. **Resolution** The feature flag was disabled globally, restoring the previous behavior where secrets are resolved during pipeline initialization. Impacted pipelines recovered immediately after rollback. **Preventive Actions** Harness is implementing the following improvements: * Enhancements to the feature flag approval workflow to more clearly display rollout scope. * Improved process and documentation for feature flags that depend on minimum delegate versions * Expanded monitoring and alerting for secret resolution failures in CI pipelines