BigLeaf experienced a minor incident on July 7, 2022 affecting Gateway Cluster: Dallas, TX and Gateway Cluster: Seattle, WA, lasting 4h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 07, 2022, 06:57 PM UTC
Today our redundant network experienced multiple events on Tunnel Endpoint Servers within our Dallas and Seattle Gateway Clusters. Customers who were connected through the affected servers automatically failed over to their secondary POP. Our engineering team was automatically alerted and immediately began working to determine the cause of these events and restore the affected servers. When these servers were brought back online, affected customer sites automatically reconnected back to their primary servers. While connected to their secondary POP, affected customers may have experienced an increase in latency and/or packet loss. Our Engineers are actively investigating the cause of these events and we will provide updates as soon as we have additional information.
- resolved Jul 07, 2022, 11:04 PM UTC
This incident has been resolved. We have performed a software update to address the issue on the affected Tunnel Endpoint servers. We will be performing an urgent maintenance window later this evening to deploy the software update to the remaining subset of Tunnel Endpoint servers as a proactive measure.