Prod2: CV is failing with errors for all customers

Update timeline

1. investigating Sep 29, 2026, 02:41 PM UTC

   We are currently investigating this issue.

2. monitoring Sep 29, 2026, 03:54 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Sep 29, 2026, 03:57 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Sep 29, 2026, 06:49 PM UTC

   This incident has been resolved.

5. postmortem Oct 02, 2026, 05:37 AM UTC

   **Summary** On September 28–29, 2026, customers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. The incident was a capacity issue. Service was restored by scaling out the affected platform component. No data loss or corruption occurred. ‌ **Customer Impact** Customers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally. ‌ **Root Cause** A large automated provisioning run created a significant number of persistent CV monitoring workers in a short window. These workers continuously poll the Continuous Verification service, and each poll requires an outbound call to validate credentials with a shared platform component. The CV service's scaled capacity was insufficient to handle this load burst. The sudden increase in concurrent outbound calls exhausted the available network connections on each service instance. Retry behavior further amplified the connection pressure, creating a feedback loop that kept the service saturated even after the initial provisioning completed. **Mitigation** The maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out completed. **Next Steps** To prevent recurrence, Harness will: * **Increase capacity headroom and add provisioning guardrails:** Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services. * **Add retry backoff:** Introduce exponential backoff and jitter on data-collection worker retries so transient failures do not immediately compound into sustained overload. * **Improve connection efficiency:** Replace the zero-idle connection configuration with a reusable connection pool to reduce overhead under high concurrency. * **Improve monitoring and alerting:** Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.