Firezone incident

GCP capacity issue

Minor Resolved View vendor source →

Firezone experienced a minor incident on July 18, 2024 affecting Relay Servers, lasting 3h 15m. The incident has been resolved; the full update timeline is below.

Started
Jul 18, 2024, 09:54 PM UTC
Resolved
Jul 19, 2024, 01:09 AM UTC
Duration
3h 15m
Detected by Pingoru
Jul 18, 2024, 09:54 PM UTC

Affected components

Relay Servers

Update timeline

  1. identified Jul 18, 2024, 09:54 PM UTC

    Google Cloud is experiencing an availability issue in multiple regions that is currently causing a slight increase in latency when establishing new Firezone connections to Resources. We are on the phone with GCP support to resolve this ASAP.

  2. identified Jul 18, 2024, 10:01 PM UTC

    We are continuing to work on a fix for this issue.

  3. monitoring Jul 18, 2024, 10:11 PM UTC

    We've rolled affected instances back and are monitoring the situation. All Firezone systems should be operational.

  4. resolved Jul 19, 2024, 01:09 AM UTC

    All services have returned to normal.