Firezone incident

Production deployed without matching published Gateways

Major Resolved View vendor source →

Firezone experienced a major incident on February 29, 2024, lasting 1d 15h. The incident has been resolved; the full update timeline is below.

Started
Feb 29, 2024, 10:00 PM UTC
Resolved
Mar 02, 2024, 01:30 PM UTC
Duration
1d 15h
Detected by Pingoru
Feb 29, 2024, 10:00 PM UTC

Update timeline

  1. resolved Mar 03, 2024, 08:17 PM UTC

    We've received reports of Gateway outages

  2. postmortem Mar 03, 2024, 08:18 PM UTC

    On 2/29/23 at 2:05pm, a production hotfix was deployed to fix an issue with IdP sync. This deployment contained backwards-incompatible changes to the Control Plane API. Typically, new Gateways are published whenever this is the case so that customers can upgrade their Gateways to avoid outages. In this case, a bug in our CI pipeline logic prevented new Gateways from being published, causing an outage that prevented existing Gateways from being able to serve Resources. This outage was resolved on 3/3/24 at 11:48pm by publishing new Gateways that are compatible with the version of the Control Plane API that was deployed. All customers are advised to upgrade their Gateways ASAP using the instructions appropriate for their deployment method, which can found at [https://www.firezone.dev/kb/administer/upgrading#upgrading-gateways](https://www.firezone.dev/kb/administer/upgrading#upgrading-gateways)