Harness incident

Feature Flag SDK authentication operations are running slow in Prod2

Minor Resolved View vendor source →

Harness experienced a minor incident on March 24, 2026 affecting Feature Flags (FF), lasting 2h 23m. The incident has been resolved; the full update timeline is below.

Started
Mar 24, 2026, 06:14 PM UTC
Resolved
Mar 24, 2026, 08:37 PM UTC
Duration
2h 23m
Detected by Pingoru
Mar 24, 2026, 06:14 PM UTC

Affected components

Feature Flags (FF)

Update timeline

  1. investigating Mar 24, 2026, 06:14 PM UTC

    We are currently investigating this issue.

  2. identified Mar 24, 2026, 06:25 PM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Mar 24, 2026, 07:03 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Mar 24, 2026, 07:03 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved Mar 24, 2026, 08:37 PM UTC

    This incident has been resolved.

  6. postmortem Mar 31, 2026, 03:31 AM UTC

    ## Summary Between **05:48 PM UTC on March 18** and **04:58 PM UTC on March 19**, two customers using [app.split.io](http://app.split.io/) with **SAML authentication using SCIM** experienced login failures. The issue was limited to customers with specific identity provisioning configurations. ## Root Cause A service update was deployed to a login-related component that changed internal request handling behavior. This caused authentication workflows for SAML-configured customers with identity provisioning enabled to fail during login processing. ## Impact * 2 customers using SAML authentication **with SCIM provisioning enabled** were unable to log in to [app.split.io](http://app.split.io/). * Customers using SAML authentication without SCIM were not affected and could log in normally. * Username and password login was unaffected. * SDKs, feature flags, and experiments continued functioning normally. * No data loss occurred. ## Remediation * Rolled back the service update that introduced the issue. * Restored SAML login for affected customers. ## Action Items To prevent such issues from happening again, we are doing/completed the following: * **Improve detection capability**: Introduced a high-priority alert to detect authentication failures sooner, enabling faster response if similar issues occur in the future. * **Reduce legacy surface area**: Continue guiding remaining customers toward the Harness Platform, where authentication flows are unified and improvements can be delivered more consistently over time.