Harness incident
Custom Dashboards are failing intermittently in prod3
Harness experienced a minor incident on May 15, 2026 affecting Custom Dashboards, lasting 1d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 15, 2026, 08:35 PM UTC
We are currently investigating this issue.
- monitoring May 15, 2026, 08:59 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring May 17, 2026, 05:36 AM UTC
We are continuing to monitor for any further issues.
- resolved May 17, 2026, 05:37 AM UTC
This incident has been resolved.
- postmortem May 28, 2026, 12:18 AM UTC
# Summary Dashboards service in Prod3 experienced intermittent failures, causing dashboards to return errors and become unavailable to some customers. ## Root Cause Analysis The incident was caused by slow Looker `search_dashboards` API calls degrading from 7 seconds to over 30 seconds, which saturated the worker thread pool and prevented the health endpoint from responding to Kubernetes liveness and readiness probes. ## Mitigation Steps Taken ### Immediate Mitigations 1. **Increased liveness probe timeout** from 15s to 30s \(failure threshold kept at 3, allowing up to 90s tolerance\) 2. **Doubled thread count** , increasing total concurrency 3. **Increased pod replica count** to maintain higher minimum availability and reduce risk of all pods becoming simultaneously unavailable ## Actions Harness will work on the following Action items to prevent recurrence. 1. **Optimize Looker SDK timeout:** Tune timeout to Looker SDK `search_dashboards` calls to prevent indefinite thread occupation 2. **Optimize Looker queries :** Investigate Looker-side query performance degradation to understand why `search_dashboards` latency increased