Harness experienced a minor incident on September 17, 2026 affecting DBDevOps and DBDevOps, lasting 1d 7h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 17, 2026, 04:57 PM UTC
We are currently investigating this issue.
- investigating Sep 17, 2026, 04:58 PM UTC
We are continuing to investigate this issue.
- identified Sep 17, 2026, 05:13 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 17, 2026, 08:05 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 17, 2026, 08:11 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 18, 2026, 02:45 PM UTC
This incident has been resolved.
- postmortem Sep 22, 2026, 07:32 PM UTC
## Summary Between approximately **8:40 AM and 1:00 PM PDT on Wednesday, September 17, 2026**, some customers running **Database DevOps** pipelines in Production experienced intermittent failures when executing schema migration steps \(for example Apply Schema and related Database DevOps steps\). Affected runs commonly surfaced a misleading error: `Invalid request: Unable to fetch port details. Please verify pipeline yaml and check if 'Enable container based execution' toggle is on for the step group.` Failures were intermittent as the same pipeline could succeed some times. Other Harness modules and non–Database DevOps CD pipelines were not broadly impacted. ## Root Cause This issue required two separate problems to occur together. First, pipeline plan creation runs on a shared pool of background worker threads, and each thread can handle work for many different customer accounts over time. The identity used to authenticate outbound service calls was tied to the thread rather than to the specific request, so a thread could occasionally carry over the identity from a previous account's request instead of using the correct one for the current request. This happened inconsistently as the same pipeline could compile fine on one run and pick up the wrong identity on the next, which is why the issue appeared and disappeared without any customer-side change, and why retries usually worked. Second, and independently, a recent change to our Database DevOps service added a stricter check to make sure the identity on an incoming request matches the account it claims to represent. Before this change, a mismatched identity like the one above would have gone unnoticed and the request would still succeed. After this change, that same mismatch was correctly rejected, but the calling service treated that rejection as a hard failure rather than retrying, which is what turned an internal identity mix-up into a visible pipeline failure for customers. Neither problem caused issues on its own. The identity mix-up was harmless until the stricter check existed to catch it, and the stricter check only became noticeable because the identity mix-up was already happening in the background ## Impact * **Product area:** Database DevOps pipeline steps \(schema apply and related migration steps\) in Production. * **Customer experience:** Intermittent pipeline failures. Some executions succeeded while others failed within minutes of each other. The reported error pointed at container/port configuration and was misleading relative to the actual cause. Retries frequently succeeded. * **Scope:** Limited to customers running Database DevOps workloads on Production clusters where the stricter account validation was active. Other CD and platform functionality was not broadly disrupted, and no data loss or unintended schema changes resulted , affected executions failed rather than applying incorrect changes. ## Remediation * **Immediate:** Rolled back the Database DevOps service change that enforced the stricter account-binding validation in the most impacted Production environment, restoring Database DevOps pipeline execution for affected customers. * **Permanent:** Shipped a fix that explicitly sets the correct source principal and restores the previous thread context afterwards around every outbound call made by Database DevOps plan creators, including the plugin capability call. Outbound calls now always carry the account context of the pipeline being compiled, regardless of which worker thread runs it. * **Verification:** Following the rollout, service logs showed no further account-mismatch rejections across monitored clusters. ## Action Items To prevent such issues from happening again, Harness will, * Enhance tests that compile plans for two accounts on a shared executor and assert each outbound call carries the correct account. * **Fail-fast on capability-check auth errors**: treat 403/auth failures as hard errors with an accurate message, instead of silently falling back to the legacy execution path. * **Alerting on account-binding rejections and plugin-capability 403 rate**: surface these signals before customer impact