Few pipelines in prod4 are running with degraded performance

Update timeline

1. investigating Jun 29, 2026, 02:56 PM UTC

   We are currently investigating this issue.

2. identified Jun 29, 2026, 02:57 PM UTC

   Harness is investigating an issue affecting our Prod 4 environments. Our log-services are affected and harness is currently working on restoring function. This can affect our CI, Chaos, IACM modules.

3. identified Jun 29, 2026, 03:02 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jun 29, 2026, 03:20 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Jun 29, 2026, 03:20 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jun 29, 2026, 03:41 PM UTC

   This incident has been resolved.

7. postmortem Jul 02, 2026, 12:25 AM UTC

   ### **Summary** During a planned security improvement to rotate service authentication secrets in the Prod 4 environment, an incorrect secret value was inadvertently applied to a subset of internal services. This resulted in authentication failures between platform components, impacting CI initialization, delegate task creation, and log streaming for certain pipeline executions. The issue was resolved by rolling back the token configuration to the previous known-good version and validating service recovery. ### **Impact** **Affected environment:** Prod 4 **Customer-visible impact:** * CI builds failed during initialization with errors such as: * _“Could not fetch token from log service”_ * _“token in request not authorized for receiving tokens”_ \(HTTP 400\) * CD and IaCM pipeline stages continued to execute but did not stream logs. * Some Shell Script steps failed during execution. * Delegate tasks could not be created or assigned for some pipeline executions. No customer data was lost or corrupted. ### **Root Cause** As part of a planned security initiative, authentication secrets used for communication between internal platform services were rotated in the Prod 4 environment. During the rotation process, an incorrect secret value was inadvertently configured for one of the services. Instead of applying the newly generated secret in the expected format, a default/incorrect value was introduced. This created a mismatch between services that authenticate using the shared token, causing authentication requests to be rejected with HTTP 400 errors. The authentication failures prevented CI components from obtaining log service tokens, disrupted delegate task creation, and prevented log streaming for affected pipeline executions. ### **Resolution** Engineering responded by: * Reverting the token configuration to the previous known-good secret. * Restoring consistent authentication across affected services. * Validating recovery through CI pipeline execution tests. * Verifying CD and IaCM pipeline execution and log streaming. * Executing a comprehensive validation suite to confirm platform functionality before closing the incident. ### **Follow-up Actions** To reduce the likelihood of similar incidents in the future, Harness will: * Implement additional pre-rotation and post-rotation validation checks to verify that the correct secret values and formats are applied before changes are activated. * Introduce automated verification of authentication between dependent services immediately following secret rotations. * Strengthen operational safeguards and deployment validation for security credential rotation procedures to detect configuration mismatches before they can impact customer workloads.