Harness experienced a minor incident on May 21, 2026 affecting Continuous Delivery - Next Generation (CDNG) and Infrastructure as Code Management (IaCM), lasting 2h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 21, 2026, 01:55 PM UTC
We are currently investigating an slowness issue in pipeline execution ui.
- investigating May 21, 2026, 02:15 PM UTC
Our investigation has revealed that the pipeline execution are happening as expected in the backend, The slowness is limited to updates in pipeline UI only.
- identified May 21, 2026, 02:33 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 21, 2026, 02:49 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 21, 2026, 04:34 PM UTC
This incident has been resolved.
- postmortem Jun 01, 2026, 08:37 AM UTC
# **Summary** On May 20 and May 21, 2026, customers on the Prod3 cluster experienced degraded pipeline performance on two separate occasions. In both cases, pipeline executions ran slower than usual and execution graph rendering was delayed because database operations were being queued on the backing MongoDB instance. Both incidents were mitigated by adjusting database write settings and adding capacity headroom, which relieved pressure on MongoDB and allowed normal operations to resume. # **Root Cause** A higher-than-usual rate of concurrent updates against a frequently modified document used during pipeline execution produced a high rate of write conflicts in MongoDB. Each conflict triggered an automatic retry, which compounded under load and drove CPU utilization on the database to a sustained high level. As CPU saturated: * MongoDB started queuing incoming commands instead of executing them immediately. * Operations dependent on those writes — pipeline step transitions and execution graph generation — slowed down or stalled while waiting for the queue to drain. # **Customer Impact** Customers on Prod3 were impacted during two windows: * May 20, 2026: ~3:15 AM to ~4:45 AM PST \(~1h 30m\). * May 21, 2026: ~6:42 AM to ~7:47 AM PST \(~1h 5m\). In both windows: * Pipeline executions ran significantly slower, with delays in step initialization and execution graph generation. * Pipelines with tight configured timeouts may have timed out as a result of the slowness. No execution failures were observed beyond timeout-related effects, and no full outage occurred. All Harness modules that rely on Pipelines on Prod3 were affected for the duration of each incident. # **Resolution** After identifying that database contention was the bottleneck, we adjusted the pipeline service's database write settings to reduce overhead on each operation and added additional database capacity to absorb the load. Together these reduced write conflicts and allowed the queued commands to drain, after which pipeline operations returned to normal. # **Prevention and Improvements** 1. Strengthen concurrency control around the high-contention update path so that concurrent updates to the same document no longer race in a way that produces write conflicts at scale. 2. Strengthen load and concurrency testing on hot execution paths so similar contention patterns are caught before they affect production traffic. 3. Improve proactive alerting on database write conflicts, queued commands, and sustained CPU utilization so the team can intervene before customer-facing latency is affected.