Firezone incident

Gateways failure to reconnect to control plane API

Notice Resolved View vendor source →

Firezone experienced a notice incident on December 23, 2025, lasting 1h. The incident has been resolved; the full update timeline is below.

Started
Dec 23, 2025, 03:50 PM UTC
Resolved
Dec 23, 2025, 04:50 PM UTC
Duration
1h
Detected by Pingoru
Dec 23, 2025, 03:50 PM UTC

Update timeline

  1. resolved Dec 25, 2025, 01:04 PM UTC

    At December 23, 3:50 PM UTC, we rolled a minor fix to our Entra authentication adapter out to production. Deploys to production are generally considered safe and are designed not to cause any packet loss or downtime for customers. Shortly after, at December 23, 4:10 PM UTC, we noticed a discrepancy in the number of nodes that successfully rejoined their control "channels" - that is, the message channel clients/gateways use to receive connection intents and configuration updates. These channels are separate from the actual WebSocket transport underlying them. The discrepancy here looks to have been very small, in the single-digit % of nodes. Upon noticing this discrepancy, we restarted API node services on each affected API node to trigger a full reset of the control plane WebSocket for these nodes. At or around December 23, 4:50 PM, UTC we verified all nodes were successfully reconnected, restoring full functionality.