JumpCloud Outage History

 **Date**: Apr 7, 2026 **Date of Incident:** Mar 30, 2026 **Description**: RCA for Directory Association Processing Delays ‌ **Summary:** Starting March 30th at approximately 15:40 MDT, JumpCloud customers experienced significant delays in directory-related updates. This included latency in password changes, user-to-group associations, and outbound provisioning reflecting in downstream systems. The root cause was identified as a specific code deployment in our Devices service that inadvertently flooded a background processing queue with unpartitioned messages, causing a bottleneck that prevented updates from processing in real-time. The issue was fully resolved by 00:25 MDT on March 31, 2026. ‌ **What Happened:** The incident was caused by a change in how the JumpCloud agent retrieves software application configurations. 1. **Traffic Spike:** The new code shifted the "source of truth" for these configurations to a new database. If a device polled the system and did not find its record in the new database, the code automatically enqueued a "track collect" request to sync the data. 2. **Unexpected Volume:** We anticipated a "lazy backfill" \(where records are created over time\), but underestimated the number of devices that had no existing software bindings. This resulted in an immediate, massive spike of nearly 280,000 messages. 3. **The Bottleneck \(Partitioning\):** Crucially, these specific messages were enqueued without a "Partition ID." In our high-scale FIFO \(First-In-First-Out\) queue architecture, messages without a partition ID are processed one-by-one rather than in parallel. This effectively "serialized" the queue, preventing us from scaling up workers to process the backlog faster and causing the observed latency. ‌ **Resolution and Recovery**: Once the offending code was rolled back, the "tap" was turned off, and no further unpartitioned messages were added to the queue. Because the bottleneck was caused by the lack of partitioning, simply scaling horizontally could not speed up the processing of the existing backlog. The team monitored the queue throughput and determined that the safest and fastest path to recovery was allowing the worker to process the existing messages sequentially rather than risking further disruption by attempting to manually manipulate the production queue. ‌ **Corrective Actions**: To ensure this type of bottleneck does not occur again, we have committed to the following: * Improving pre-production testing to better simulate the scale and conditions that can occur in production queue processing * Reviewing other areas of the platform where similar patterns could produce unexpected request spikes * Enhancing monitoring and alerting thresholds to enable faster detection and response when queue backlogs begin to form * Strengthening our deployment validation process to more thoroughly account for background data migrations before releasing dependent code changes

 ‌ **Date**: Mar 17, 2026 **Date of Incident:** Mar 12, 2026 **Description**: RCA for Agent Backend \(HAProxy\) System Degradation ‌ **Summary:** On March 12, 2026, from 10:05 AM to 2:45 PM MDT, JumpCloud experienced a significant service degradation affecting Agent-related activities. During this window, agent updates, including syncing users, passwords, policies and other agent data, as well as new agent installations were unavailable. This was caused by a "thundering herd" event triggered by a backend traffic-shaping change. We have since identified the root causes and implemented infrastructure changes to prevent a recurrence. ‌ **What Happened?** At 10:00 AM MDT, our engineering team enabled a feature flag \(a "circuit breaker"\) designed to protect our System Insights API from high load by returning `503 Service Unavailable` responses for certain non-critical requests. While the flag performed its intended function, it had an unforeseen secondary effect on the JumpCloud Agent’s connection logic. Because the agents could not reuse existing connections for these specific failed requests, hundreds of thousands of agents in our main production environment attempted to establish new mTLS \(mutual TLS\) connections simultaneously. This created a "Thundering Herd" event that saturated our HAProxy ingress layer, exhausting CPU resources and causing a cascade of connection failures. ‌ **Root Cause:** The prolonged nature of this incident was the result of three distinct, overlapping bottlenecks that our team had to isolate and resolve one by one: 1. **CPU-Intensive SSL Handshaking:** Establishing an mTLS connection is a CPU-intensive process. The sheer volume of simultaneous connection attempts pushed our HAProxy pods to their resource limits. This caused the pods to become unresponsive, leading to "Out of Memory" \(OOM\) kills and failed health probes. 2. **Health Check Death Spiral:** Our internal health checks initially relied on a Layer 7 SSL validation. Because the CPU was 100% occupied with agent reconnections, the pods couldn't respond to their own health checks in time. This caused the system to erroneously mark healthy pods as "down”, removing them from the rotation and further overwhelming the remaining pods. 3. **Load Balancer Handshake Saturation:** As we attempted to scale our infrastructure, the Application Load Balancer \(ALB\) encountered a throughput bottleneck specifically related to the rate of new connection establishments. The surge of agents attempting to negotiate new SSL handshakes at the same time exceeded the ALB's burst capacity, temporarily preventing even healthy backend pods from receiving and processing traffic. ‌ **Why It Took Time to Resolve:** While reverting the flag was the correct first step, the agents were already in an aggressive retry loop that continued even after the 503 errors stopped. We had to experiment with several configurations \(adjusting health check intervals and timeout windows\) to find a balance that allowed pods to stay "alive" long enough to process the backlog. Stability was achieved only once we implemented Concurrency Control. By lowering the maximum allowed concurrent connections per pod, we stopped the CPU from over-committing to handshakes, allowing the system to reliably process a controlled flow of traffic until the global queue cleared. ‌ **Corrective Actions / Risk Mitigation:** **1.\) Edge Infrastructure Hardening** We are standardized on a new high-availability configuration for our HAProxy ingress layer. * **Concurrency Governance**: We have implemented a strict maxconn limit per pod. This acts as a "pressure valve," ensuring that the CPU remains available to process existing requests rather than becoming saturated by new connection attempts. * **Dynamic Capacity Management via Autoscaling**: We are implementing Horizontal Pod Autoscaling \(HPA\) for our HAProxy ingress layer, calibrated to trigger based on both CPU utilization and active connection counts. This ensures we can absorb sudden traffic fluctuations and also maintain a controlled flow of requests to our backend services. **2.\) Agent Connectivity Optimization** We are updating the JumpCloud Agent’s communication layer to be more "network-aware" during degraded states: * **Enhanced Connection Pooling**: We are reconfiguring the agent's HTTP transport logic to maximize the reuse of existing idle connections. This significantly reduces the "Connection Tax" on our backend during high-traffic events. * **Streamlined Resource Handling**: We are implementing stricter protocols for draining and closing HTTP response bodies, ensuring that pooled connections are returned to the rotation immediately and reliably. **3.\) Adaptive Retry Logic \(Jitter\)** To further break up "synchronized" traffic spikes: * **Introduction of Jitte**r: While our agents currently use exponential backoff for poll requests, we are adding randomized "jitter" to our retry intervals. This spreads reconnection attempts across a wider window, preventing large blocks of agents from hitting the service at the exact same millisecond. * **Standardizing Resilient Retry Logic:** We are transitioning the Agent’s default HTTP client to a unified **exponential backoff** model for all request types. * **Controlled Rollou**t: This update will be managed via a staged rollout to monitor for any unforeseen side effects on fleet-wide connectivity patterns.