Some customers in Prod1 are experiencing issues with secret evaluations.

Update timeline

1. investigating Feb 16, 2026, 07:37 PM UTC

   We are currently investigating this issue.

2. identified Feb 16, 2026, 07:41 PM UTC

   The issue has been identified and we are rolling out mitigations.

3. monitoring Feb 16, 2026, 07:58 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 16, 2026, 08:09 PM UTC

   This incident has been resolved.

5. postmortem Mar 02, 2026, 05:20 PM UTC

   ## **Summary** On **February 16, 2026, at 7:04 PM UTC** , an incident occurred that caused pipeline execution failures for workflows using cross-scope secrets. Affected customers were unable to execute pipelines that referenced secrets stored in a different scope than the pipeline’s execution scope.‌ The issue was introduced as part of a recent production changes. The system was rolled back to the previous stable release to restore normal functionality.‌ ## **Root Cause** A regression introduced in a recent release impacted the secret resolution flow for cross-scope secret managers.‌ Under specific conditions, the system incorrectly resolved the scope of the secret manager during decryption, causing the lookup to fail. As a result, pipelines referencing cross-scope secrets were unable to complete secret evaluation successfully.‌ The issue was not detected prior to production deployment due to a gap in automated test coverage for cross-scope decryption scenarios. ## **Impact** Customers using cross-scope secrets experienced pipeline execution failures. Pipelines that relied on secrets stored in a different scope than the pipeline execution scope were unable to run successfully during the impact window. Other pipeline executions that did not use cross-scope secret managers were unaffected. ‌ ## **Mitigation**‌ **Immediate** ‌ * Rolled back the production deployment to the previous stable release. * Functionality was fully restored following the rollback. ### **Permanent** * Enhanced test coverage to include cross-scope secret evaluation scenarios. * Updated automation to validate pipeline execution using non-default Secret Managers. * Reviewed and improved alerting for error rate monitoring related to bulk encryption and secret resolution flows. * Reviewed deployment rollback processes to prevent similar environment inconsistencies. * Strengthened code review validation for scope-sensitive changes. ## **Action Items** To prevent such items in the future * Add comprehensive automation coverage for cross-scope secret decryption workflows. * Enhance and further solidify CI/CD validation for non-default Secret Manager scenarios. * Improve monitoring and alerting for secret resolution error rates.