Code Module is not accessible on prod1/2/3

Update timeline

1. investigating Mar 05, 2026, 10:16 PM UTC

   We are currently investigating this issue.

2. identified Mar 05, 2026, 10:17 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 05, 2026, 10:17 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 05, 2026, 11:42 PM UTC

   This incident has been resolved.

5. postmortem Mar 09, 2026, 08:33 PM UTC

   ## Summary Between **4:20 PM and 5:16 PM EST on Thursday, March 5, 2026**, customers using the **Harness Code modules** experienced a production outage in Harness production clusters **Prod1, Prod2, and Prod3**. Git repositories were unreachable during this outage. ## Root Cause We experienced a surge in metrics that overwhelmed the metric collectors on the Kubernetes pods. As a result, the Git pods were impacted. The StatefulSet became unschedulable, and resizing of the metric collectors was required to remedy the situation. ## Impact All code repositories were offline during the event across all three production clusters. ## Remediation Engineering increased the memory allocated to the metric collectors and redeployed the configuration. After redeployment, the Git pods were rescheduled and service was restored. ## Action Items To prevent such issues from happening, we are implementing the following: * **Enhance monitoring and alerting** – Add health monitors for metric-gathering collectors and rebalance metric growth across the cluster. * **Review capacity planning** – Proactively monitor metric collector usage and scale them appropriately with sufficient headroom to handle spikes.