Firezone Outage History

Load balancers have terminated all stale connections and all up-to-date Firezone Gateways should be now be reconnected.

This incident has been resolved.

The issue turned out to be a misbehaving NAT device between some customer Gateways and our Relay servers and not an issue in Firezone's infrastructure specifically.

This incident has been resolved.

The fix has been deployed.

The load balancer has been updated. All services should be back to operational.

At December 23, 3:50 PM UTC, we rolled a minor fix to our Entra authentication adapter out to production. Deploys to production are generally considered safe and are designed not to cause any packet loss or downtime for customers. Shortly after, at December 23, 4:10 PM UTC, we noticed a discrepancy in the number of nodes that successfully rejoined their control "channels" - that is, the message channel clients/gateways use to receive connection intents and configuration updates. These channels are separate from the actual WebSocket transport underlying them. The discrepancy here looks to have been very small, in the single-digit % of nodes. Upon noticing this discrepancy, we restarted API node services on each affected API node to trigger a full reset of the control plane WebSocket for these nodes. At or around December 23, 4:50 PM, UTC we verified all nodes were successfully reconnected, restoring full functionality.

Incident has been resolved.

The Firezone control plane API briefly stopped accepting new client connections due to a database indexing job that took longer than expected. All services are now fully online.

This incident has been resolved.

The incident has been resolved and all Relay servers are confirmed to be functioning as normal.

This incident has been resolved.

On July 21, at around 5:30a pacific time, database maintenance was performed to remove stale policy authorization records. Due to a bug in the event processing handler for each connected client, this resulted in large numbers of policy updates to be sent to some Firezone clients that were connected at the time. Depending on the level of resource availability on the host machine at the time of occurrence, some Firezone clients were unable to process this burst of events, and subsequently started dropping them. In some cases, this could have led to one or more Resource(s) being unavailable in the client that the user would normally have access to. Once the client reconnected, either through a client restart or re-authentication, the list of Resources would be reset, resolving the issue. A fix for the event processing filter was deployed later this day at around 5:00 pm pacific time, mitigating the potential for similar issues to occur in the future.

The Firezone service is now fully-functional.

The incident has been resolved.