Firezone experienced a major incident on December 30, 2025 affecting Control Plane API, lasting 1h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 30, 2025, 02:12 PM UTC
We are currently investigating this issue.
- monitoring Dec 30, 2025, 02:22 PM UTC
We are receiving alerts for intermittent connection timeouts and sign in issues to the control plane API. We are investigating the source of these alerts.
- monitoring Dec 30, 2025, 02:22 PM UTC
The source of the issue has been identified: a very large number of devices are attempting to connect to the control plane API at the same time. We are currently scaling up capacity to meet demand.
- resolved Dec 30, 2025, 02:28 PM UTC
Issue has been resolved.
- postmortem Dec 30, 2025, 02:32 PM UTC
**Incident Postmortem: Control Plane Connectivity Disruption** **Date:** December 30, 2025 **Impact:** Components \(clients and Gateways\) running in customer environments experienced repeated disconnections and, in some cases, did not automatically recover. On December 30, 2025, Firezone experienced a control-plane connectivity disruption that caused Clients and Gateways to repeatedly reconnect and, in some cases, remain offline. The issue was caused by an unexpected interaction between a backend schema change, retry behavior in Clients/Gateways, and load balancer rate limiting. A backend change modified the metadata exchanged between Firezone services and connected components. Older connection processes correctly detected the change and restarted in order to reconnect using the new format. Under certain conditions, Clients and Gateways interpreted these restarts as successful connections followed immediately by disconnects, which caused retry logic to occur repeatedly instead of backing off as intended. This resulted in components reconnecting far more aggressively than intended. As reconnect attempts increased, traffic to the control plane spiked and load balancer rate limits began returning HTTP 429 responses to stabilize the system. Some components incorrectly treated these responses as fatal errors, preventing recovery and leaving a subset of components offline until either re-authenticated or restarted. The incident was mitigated by enforcing rate limits at the load balancer using HTTP response codes Clients and Gateways recognize for retry-able failures, resulting in proper backoff behavior. To prevent similar incidents, we’re implementing the following fixes: * Update Client and Gateway retry behavior to handle more classes of transient network issues so that they in effect are always retried for a longer period of time, healing broken control plane connections. * Improving our staging and test environments with more realistic load simulation. We thoroughly test each change in a staging environment before rolling out to production, but this specific combination of error cases was not covered. * Further tune load balancer configuration to catch these reconnect storms much more quickly while still allowing legitimate traffic through.