Harness incident
Some customers in Prod1 are experiencing issues with secret evaluations.
Harness experienced a minor incident on February 16, 2026 affecting Continuous Delivery - Next Generation (CDNG) and Continuous Integration Enterprise(CIE) - Mac Cloud Builds and 1 more component, lasting 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 16, 2026, 07:37 PM UTC
We are currently investigating this issue.
- identified Feb 16, 2026, 07:41 PM UTC
The issue has been identified and we are rolling out mitigations.
- monitoring Feb 16, 2026, 07:58 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 16, 2026, 08:09 PM UTC
This incident has been resolved.
- postmortem Mar 02, 2026, 05:20 PM UTC
## **Summary** On **February 16, 2026, at 7:04 PM UTC** , an incident occurred that caused pipeline execution failures for workflows using cross-scope secrets. Affected customers were unable to execute pipelines that referenced secrets stored in a different scope than the pipeline’s execution scope. The issue was introduced as part of a recent production changes. The system was rolled back to the previous stable release to restore normal functionality. ## **Root Cause** A regression introduced in a recent release impacted the secret resolution flow for cross-scope secret managers. Under specific conditions, the system incorrectly resolved the scope of the secret manager during decryption, causing the lookup to fail. As a result, pipelines referencing cross-scope secrets were unable to complete secret evaluation successfully. The issue was not detected prior to production deployment due to a gap in automated test coverage for cross-scope decryption scenarios. ## **Impact** Customers using cross-scope secrets experienced pipeline execution failures. Pipelines that relied on secrets stored in a different scope than the pipeline execution scope were unable to run successfully during the impact window. Other pipeline executions that did not use cross-scope secret managers were unaffected. ## **Mitigation** **Immediate** * Rolled back the production deployment to the previous stable release. * Functionality was fully restored following the rollback. ### **Permanent** * Enhanced test coverage to include cross-scope secret evaluation scenarios. * Updated automation to validate pipeline execution using non-default Secret Managers. * Reviewed and improved alerting for error rate monitoring related to bulk encryption and secret resolution flows. * Reviewed deployment rollback processes to prevent similar environment inconsistencies. * Strengthened code review validation for scope-sensitive changes. ## **Action Items** To prevent such items in the future * Add comprehensive automation coverage for cross-scope secret decryption workflows. * Enhance and further solidify CI/CD validation for non-default Secret Manager scenarios. * Improve monitoring and alerting for secret resolution error rates.