Harness incident

Scheduled change request delayed for a subset of flags

Minor Resolved View vendor source →

Harness experienced a minor incident on May 2, 2026, lasting —. The incident has been resolved; the full update timeline is below.

Started
May 02, 2026, 12:01 AM UTC
Resolved
Apr 28, 2026, 05:21 PM UTC
Duration
Detected by Pingoru
May 02, 2026, 12:01 AM UTC

Update timeline

  1. resolved May 02, 2026, 12:01 AM UTC

    Between April 28 and May 1, 2026, a subset of scheduled change requests failed to execute at their scheduled times. The issue was introduced by a production deployment on April 28 and fixed with a new deployment on May 1. All affected scheduled jobs that had not been manually withdrawn by customers have since been executed successfully.

  2. postmortem May 06, 2026, 03:58 AM UTC

    ## Summary Between April 28 and May 1, 2026, a subset of scheduled change requests failed to execute at their scheduled times. The issue was introduced by a service update on April 28 and resolved on May 1. All affected scheduled jobs that had not been manually withdrawn by customers have since been executed successfully. ## Root Cause A service update exposed a latent bug in the path that retrieves applicable governance policies for a change request. This caused scheduled change request execution to fail, leaving affected change requests stuck in their scheduled state with no automatic retry. ## Impact * Scheduled change request execution failed silently for a subset of flags on affected accounts * Affected change requests were not applied at their scheduled time; they remained visible as scheduled in the UI * SDKs and runtime flag evaluation were not impacted * No data loss occurred; all backlogged jobs have since been processed ## Timeline | Time \(UTC\) | Event | | --- | --- | | Apr 28, 18:21 | Production deployment introduced the regression | | Apr 28, 20:31 | First scheduled change request execution fails | | May 1, 08:35 | First customer report received | | May 1, 09:43 | Escalated to engineering | | May 1, 19:43 | Fix deployed to production | | May 1, 22:28 | All stuck scheduled jobs confirmed published | ## Remediation * Deployed a forward fix addressing the underlying latent bug and adding additional protections * Reset scheduler tasks, allowing the scheduler to re-execute them; all have since been confirmed published ## Action Items * Enhance test coverage for scheduled change request execution edge cases * Add alerting for scheduled changes that have not executed within a threshold of their scheduled time, enabling proactive detection ahead of customer reports