Hosted CI customers using secure connect are facing connectivity issues

Update timeline

1. investigating May 12, 2026, 09:38 PM UTC

   We are currently investigating this issue.

2. identified May 12, 2026, 09:40 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 12, 2026, 10:50 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 12, 2026, 11:10 PM UTC

   This incident has been resolved.

5. postmortem May 20, 2026, 01:56 AM UTC

   ## Summary On May 12, 2026, Harness's Secure Connect service experienced a significant disruption that affected customers using Secure Connect for git connector checks and CI pipeline execution. Affected customers experienced connection timeouts and failures when attempting to use Secure Connect-enabled connectors, while connectors operating without Secure Connect continued to function normally. The disruption was caused by an automated infrastructure maintenance event in our Google Kubernetes Engine \(GKE\) environment that left a critical load balancer in an unreachable state. The issue was resolved by recreating the affected load balancer and updating the corresponding DNS record to restore connectivity. **Impact:** * Secure Connect-enabled git connector tests failing with connection timeouts * CI pipelines hanging or failing when Secure Connect was enabled * Connectors operating without Secure Connect were unaffected ## Root Cause The disruption was triggered by an automated GKE \(Google Kubernetes Engine\) control plane upgrade on the cluster hosting our internal Secure Connect FRPS \(Fast Reverse Proxy Server\) infrastructure. The upgrade executed in two sequential phases in rapid succession. Each phase restarts the GKE cloud controller manager, which is responsible for managing the lifecycle of Kubernetes LoadBalancer service IP addresses. As part of its normal reconciliation process, the controller performs a delete-then-re-insert cycle on IP address reservations when it restarts. The second phase of the upgrade began before the first reconciliation cycle had fully completed. This created a race condition where the controller deleted the IP address reservation for the internal FRPS load balancer but was interrupted before it could re-acquire it. No Harness application deployments caused or contributed to the issue. ‌ ## Mitigation The following steps were taken to restore service: 1. **FRPS pod restart** — An initial rollout restart of the FRPS deployment was performed. This restored client connectivity to the FRPS server but did not resolve the underlying IP orphaning issue. 2. **Load balancer recreation** — The internal Kubernetes service \(`frps-internal`\) was deleted and recreated, triggering GKE to provision a new load balancer with a new IP address 3. **DNS record update** — The Cloud DNS A record for was updated to point to the new load balancer IP. This restored end-to-end routing for all Secure Connect traffic. Full connectivity was confirmed via internal testing and subsequently verified by affected customers. ## Preventative Actions Harness is committed to preventing this class of incident from recurring. The following actions are being implemented: