Hosted CI mac builds are impacted

Update timeline

1. investigating Feb 24, 2026, 04:37 PM UTC

   We are currently investigating this issue.

2. identified Feb 24, 2026, 04:46 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 24, 2026, 05:01 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 24, 2026, 05:11 PM UTC

   This incident has been resolved.

5. postmortem Mar 04, 2026, 01:00 AM UTC

   ## **Summary** On February 24, 2026, Harness experienced an incident that temporarily affected **Mac Cloud build scheduling across production environments**. During the incident window, new Mac build jobs could not be scheduled due to an orchestration control-plane disruption. The issue was detected immediately through monitoring alerts, investigated by the on-call engineering team, and resolved after restoring the affected control-plane node. Full service was restored shortly thereafter. ## **Root Cause** The incident occurred due to a loss of quorum in the orchestration control plane responsible for scheduling Mac builds. This was due to two separate issues of memory pressure and a traffic spike at the same time.This resulted in a temporary interruption to Mac build scheduling until the degraded node was restored and quorum was re-established. ## **Impact** * **Affected Service:** Mac Cloud builds * **Affected Regions:** Production clusters \(Prod1, Prod2, Prod3, Prod4\) * **Customer Impact:** * New Mac build jobs could not be scheduled during the incident window * Existing running builds were not impacted * **Services Not Impacted:** * Linux Cloud builds * Windows Cloud builds * Self-hosted build infrastructure * Other Harness CI/CD services \(pipelines, artifacts, deployments\) **Mitigation** Engineering teams restored the degraded orchestration node, allowing the cluster to re-establish quorum and elect a leader. Once the leader election was completed, Mac build scheduling resumed and services returned to normal operation. ## **Prevention and Improvements** To reduce the likelihood of similar incidents in the future, the following actions are being implemented: * Reducing scheduling load on the orchestration layer by optimizing infrastructure in terms of reliability. * Improving monitoring and health checks for control-plane nodes.