BigLeaf incident

San Jose POP: Isolated Server Failure

Notice Resolved View vendor source →

BigLeaf experienced a notice incident on April 24, 2022 affecting Gateway Cluster: San Jose, CA, lasting 21h 17m. The incident has been resolved; the full update timeline is below.

Started
Apr 24, 2022, 07:58 PM UTC
Resolved
Apr 25, 2022, 05:16 PM UTC
Duration
21h 17m
Detected by Pingoru
Apr 24, 2022, 07:58 PM UTC

Affected components

Gateway Cluster: San Jose, CA

Update timeline

  1. monitoring Apr 24, 2022, 07:58 PM UTC

    Today, our redundant network experienced an isolated server failure within our San Jose (SJC) POP, from ~06:21 - 06:29 GMT. Customers connected through the affected server were automatically transitioned to their secondary POP, and reconnected back to their primary server once it was restored. Our engineering team was automatically alerted and began investigating the issue. As a precautionary measure, we are redistributing customers with primary tunnels on the affected server to other tunnel servers within the POP. During this event, customers with primary tunnels on the affected server may have experienced a slight increase in latency and/or packet loss during the failure, and may experience a similar impact again when their tunnel is migrated to a new primary server. Our engineering team is continuing to investigate root cause.

  2. resolved Apr 25, 2022, 05:16 PM UTC

    This incident has been resolved.