Harness experienced a minor incident on July 17, 2026 affecting Continuous Integration Enterprise(CIE) - Self Hosted Runners and Continuous Integration Enterprise(CIE) - Mac Cloud Builds and 1 more component, lasting 4h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 17, 2026, 05:16 PM UTC
We are currently investigating this issue.
- identified Jul 17, 2026, 05:34 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 17, 2026, 06:12 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 17, 2026, 09:56 PM UTC
This incident has been resolved.
- postmortem Aug 04, 2026, 02:48 AM UTC
# Summary On 17 July 2026, following a routine code deployment, customers on older delegate versions \(858xx and below\) began experiencing delayed CI builds on Harness Cloud-hosted builds using our global build-queueing capability. Affected builds experienced an unexpected pause of up to approximately 8 minutes at the "waiting for infrastructure" stage before continuing, rather than proceeding within the expected sub-second time. Overall build slowness was intermittent. # Impact * All CI builds were potentially subject to delay; impact was most pronounced for builds on Harness Cloud-hosted infrastructure using the global build-queueing feature. * Affected builds experienced an unexplained pause of up to approximately 8 minutes before continuing, followed by a slower "cold start" since a pre-reserved compute slot was not available — this presented to users as slow builds rather than build failures. * Accounts running on newer delegate versions \(858xx and above\) were not impacted. * No builds failed outright as a direct result of this issue, and no data was lost. # Root Cause The root cause was an internal code change that inadvertently broke how a specific build-queueing record was read back from our database once builds that had already been queued under the previous version of the code encountered the newly deployed version. We resolved the immediate impact by cleaning up the affected records and reverting the underlying code change, and we are implementing several safeguards to prevent this class of issue from recurring. # Next Steps We assess the risk of a similar recurrence as low as the following actions are being understaken.The specific code path that caused this incident has already been reverted, and we are implementing structural safeguards so that this general class of issue cannot recur, regardless of where in the codebase it might otherwise occur. | **Corrective / Preventive Action** | | --- | | Add explicit, stable identifiers to all internal data classes that get stored in our database, so that future internal code reorganizations cannot break the system's ability to read back previously stored records. | | Introduce rollback and backward-compatibility testing in our pre-production environment, specifically designed to catch this class of issue before it reaches production. |