Firezone incident
Clients unable to make new connections to Resources
Firezone experienced a major incident on April 3, 2024 affecting Control Plane API and Admin Portal and 1 more component, lasting 8h 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 03, 2024, 05:54 PM UTC
We're investigating an incident with the latest release of the Firezone Gateway (1.0.0-pre.12). We advise all customers to not upgrade their Gateways at this time if they have not done so. We will give a status updates here as we discover more information.
- investigating Apr 03, 2024, 09:09 PM UTC
We are continuing to investigate the issue. No new information to report at this time.
- identified Apr 04, 2024, 01:39 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 04, 2024, 01:40 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Apr 04, 2024, 02:48 AM UTC
This incident has been resolved.
- postmortem Apr 04, 2024, 03:21 PM UTC
On 4/3/24 around 8:30a pacific, a new version of the Firezone Relays was deployed to production which contained a logic bug that caused them to fail to be used by Clients after a certain period of time. This logic bug was not caught in automated testing or our staging environment because it only occurred in Relays with a certain access pattern. We rolled back the afflicted Relays around 5pm pacific time which resolved the issue. We’ve since added better load testing to catch issues like this in the future.