Twingate Outage History

Started Jul 08, 2026, 10:04 PM UTC· resolved Jul 09, 2026, 12:24 AM UTC

1. investigating Jul 08, 2026, 10:04 PM UTC

   Our upstream provider for Linux packages is having an issue impacting Twingate's Linux package downloads.

2. investigating Jul 08, 2026, 10:06 PM UTC

   We are continuing to investigate this issue.

3. investigating Jul 08, 2026, 10:07 PM UTC

   We are continuing to investigate this issue.

4. investigating Jul 09, 2026, 12:24 AM UTC

   We are continuing to investigate this issue.

5. resolved Jul 09, 2026, 12:24 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 28, 2026, 09:24 AM UTC· resolved May 28, 2026, 05:27 PM UTC

1. investigating May 28, 2026, 09:24 AM UTC

   We are currently investigating this issue.

2. investigating May 28, 2026, 09:25 AM UTC

   We are continuing to investigate this issue.

3. investigating May 28, 2026, 10:55 AM UTC

   We are continuing to investigate this issue.

4. investigating May 28, 2026, 10:57 AM UTC

   We are continuing to investigate this issue.

5. monitoring May 28, 2026, 12:02 PM UTC

   The issue has been fixed. We are still working with cloud provider on root cause.

6. monitoring May 28, 2026, 01:47 PM UTC

   We're continuing to work with our cloud provider to identify the root cause.

7. monitoring May 28, 2026, 04:22 PM UTC

   We're continuing to work with our cloud provider to identify the root cause.

8. resolved May 28, 2026, 05:27 PM UTC

   We are currently awaiting the completion of our cloud provider’s investigation into the issue. In the meantime, we continue to closely monitor the system, and all service metrics have returned to normal operating levels. At this time, we are closing this incident. We will publish a comprehensive incident report as soon as it becomes available.

9. postmortem Jun 03, 2026, 04:39 PM UTC

   # Incident Report – Authorization Service Degradation ## Components Impacted Control Plane – Authorization Service ## Summary On May 28, 2026, between 10:00 UTC and 12:30 UTC, Twingate experienced a degradation of its Authorization service that affected approximately 15% of active connections at peak impact. During the incident, authorization requests experienced elevated network latency. As request processing times increased, the Authorization service's effective capacity to handle incoming traffic was reduced, resulting in elevated error rates and intermittent authorization failures for a subset of customers. Twingate operates the Authorization service across multiple cloud regions in an active-active configuration. While the platform remained available throughout the event, the combination of increased request latency and reduced service capacity led to customer impact until additional capacity was provisioned and service performance stabilized. ## Root Cause The incident was triggered by elevated network latency affecting communication paths used by the Authorization service. As requests took longer to complete, individual service instances were able to process fewer requests than normal. This reduction in throughput exposed a limitation in our auto-scaling configuration, which primarily relied on CPU utilization to determine service capacity requirements. As request-processing workers spent more time waiting on network operations, CPU utilization declined even as request latency increased. As a result, the service scaled down during a period of elevated request latency, reducing available capacity and amplifying customer impact. Recovery efforts were further complicated by an unusually high rate of spot instance preemptions in two regions, which reduced available compute capacity during stabilization. ## Resolution Engineering teams mitigated the incident by manually increasing Authorization service capacity and expanding available cluster resources. As additional capacity came online, request latency and error rates returned to normal levels and service performance fully recovered. ## Corrective Actions ### Completed * Increased the baseline capacity of the Authorization service by raising the minimum number of service instances. * Increased baseline node capacity across affected Kubernetes clusters. * Implemented additional rate controls for unusually large authorization requests to better protect overall service availability during periods of elevated load. ### In Progress * Rebalance spot and on-demand node capacity to reduce sensitivity to spot instance interruptions. * Enhance auto-scaling policies to incorporate latency and service performance metrics in addition to CPU utilization. * Expand monitoring and alerting to better detect conditions where request latency increases while resource utilization decreases.

Read the full incident report →

Started May 08, 2026, 08:44 AM UTC· resolved May 08, 2026, 10:55 AM UTC

1. investigating May 08, 2026, 08:44 AM UTC

   We are seeing an issue with relay443. We are investigating. Note: These are special relays that offer connectivity over port 443 and are used by a subset of customers. So majority of customers shouldn't be seeing an issue,

2. identified May 08, 2026, 10:28 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 08, 2026, 10:55 AM UTC

   The fix has been rolled out and relays443 are now fully functional.

4. postmortem May 14, 2026, 09:19 AM UTC

   **Components impacted** * Data Plane - Relay 443 flows only **Summary** On May 9, 2026, a subset of Twingate customers experienced an interruption to network access affecting Relay 443 connectivity. The disruption lasted approximately 2 hours and 10 minutes and was caused by a configuration error introduced during a routine software update. We have resolved the issue and are taking concrete steps to prevent a recurrence. **Root Cause** Twingate regularly releases software updates to our relay infrastructure, deploying them in stages across groups to minimize risk. As part of this update cycle, we also migrated our deployment tooling to a newer version of our infrastructure management system. The newer tooling applies stricter configuration validation rules than its predecessor. While preparing the update, our team identified and corrected a configuration conflict this stricter validation had surfaced. However, the fix was shipped in the same release bundle as a second change that was designed to serve as a prerequisite — specifically, a change that prevents the stricter validation from modifying existing, live deployments. The result was that a critical network configuration — the definition that tells our relay nodes to accept connections on port 443 — was silently removed from existing cluster deployments. **Why wasn't the issue detected earlier with the initial rollout?** Our continuous smoke testing did not include end-to-end coverage for Relay 443 specifically, so no automated alert fired when previous groups were deployed. Because those early upgrade groups carried relatively low Relay 443 traffic, no customer-visible impact surfaced either, leaving the issue undetected for approximately 48 hours until the higher-traffic groups were rolled out. **Why wasn't the issue caught in lower environments?** The issue was identified in a lower environment and a fix was prepared. However, the fix was deployed in the same release as the problematic change rather than as a prerequisite, which prevented it from taking full effect on existing cluster upgrades. Subsequent deployments confirmed the issue was fully resolved. **Remediation** Once we confirmed the root cause, we re-deployed the corrected relay configuration — explicitly restoring the port 443 host mapping — across all cluster groups in sequence. We validated each group before proceeding to the next. Full recovery was confirmed at 11:55 UTC on May 9. **Corrective actions** Short-term: * **Completed:** Redeployed to production to confirm the fix is stable and the issue will not recur. * We are adding regional smoke tests for Relay 443 to match the coverage already in place for other relay deployments, ensuring failures like this are caught before impacting customers.

Read the full incident report →

Started Oct 20, 2025, 03:49 PM UTC· resolved Oct 20, 2025, 10:45 PM UTC

1. monitoring Oct 20, 2025, 03:49 PM UTC

   The AWS outage is affecting GemFury, a service we use to host our APT/RPM packages.

2. monitoring Oct 20, 2025, 05:02 PM UTC

   Our testing indicates that APT/RPM packages are now operational, though Gemfury continues to report degraded performance. We will continue to monitor.

3. resolved Oct 20, 2025, 10:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 12, 2025, 06:18 PM UTC· resolved Jun 12, 2025, 11:49 PM UTC

1. investigating Jun 12, 2025, 06:18 PM UTC

   We are currently investigating service issues related to our cloud provider. The disruption is affecting the Twingate service, including the Admin Console, Report Downloads, and Network Dashboards. We will continue to monitor the situation and provide updates as we learn more.

2. identified Jun 12, 2025, 07:15 PM UTC

   Our cloud provider has acknowledged a service issue. We are continuing to monitor the situation and will provide updates as we learn more.

3. monitoring Jun 12, 2025, 08:49 PM UTC

   Our cloud provider has indicated that their service is recovering. Impacted Twingate services are now back online, but we are currently processing a backlog of Events. We expect this to be fully resolved within the next few hours and will provide an update when processing is complete.

4. resolved Jun 12, 2025, 11:49 PM UTC

   Our cloud provider has confirmed full recovery of their services, and our backlog of Events has now been fully processed. We consider this incident fully resolved.

Read the full incident report →

Started May 28, 2025, 08:53 PM UTC· resolved May 28, 2025, 09:32 PM UTC

1. identified May 28, 2025, 08:53 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring May 28, 2025, 09:16 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 28, 2025, 09:32 PM UTC

   This incident is resolved now. We'll provide a post-mortem as soon as we have details.

4. postmortem Jun 16, 2025, 06:26 PM UTC

   ### **Incident Duration** **UTC Time**: May 28, 2025, from **20:45 to 21:14** \(29 minutes\) ### **Components Impacted** **Control Plane Services** * Authentication \(Enterprise & Social\) * Multi-Factor Authentication \(MFA\) * Authorization * Connector Heartbeat **Management Plane Services** * Identity Provider Sync * Public API * Admin Console * Real-Time Updates * Client Log Upload * 3rd Party Integrations * Network Dashboards * DNS Filtering Dashboards ### **Summary** On May 28, 2025, Twingate experienced a service disruption that impacted multiple core services for approximately 29 minutes. The issue stemmed from a misconfiguration introduced during a rollout that updated routing rules in our global load balancer. A new routing rule was unintentionally prioritized, resulting in most API traffic being directed to a single backend. This overwhelmed the target service, leading to elevated error rates. Lower environments did not expose this issue due to their reduced traffic levels. The incident was promptly detected and the rollback was initiated quickly. However, recovery was delayed due to infrastructure dependencies that also relied on affected components. A manual intervention using local tooling ultimately restored service. ### **Root Cause** A routing misconfiguration caused a traffic imbalance that overwhelmed a backend service, leading to widespread API failures. ### **Resolution Timeline** * **20:45 UTC** – Incident begins. API traffic misrouted. * **20:48 UTC** – Incident detected and triage begins. * **20:52 UTC** – Rollback initiated. * **21:05 UTC** – Automated rollback process encounters delays. * **21:09 UTC** – Engineers execute manual fix. * **21:14 UTC** – Services fully recovered. ### **Corrective and Preventative Actions** ### **Short-Term \(Completed\)** * ✅ Routing rule corrected. * ✅ Alerts added to detect backend traffic anomalies in lower environments. * ✅ Improved resource scaling thresholds. ### **Mid-Term \(In Progress\)** * ⚙️ Decouple infrastructure tooling from service dependencies to ensure faster recovery options. ### **Long-Term \(Planned\)** * 🔄 Introduce backend sharding to isolate customer workloads and reduce incident impact. We sincerely apologize for the disruption and are taking actions to improve our systems' resilience. We appreciate your continued support and trust.

Read the full incident report →

Started Apr 02, 2025, 04:19 PM UTC· resolved Apr 02, 2025, 05:25 PM UTC

1. identified Apr 02, 2025, 04:19 PM UTC

   We are seeing higher latency than usual with our cloud provider's APIs to process data used for our dashboards and reports. They confirmed the issue with their service. We hope this should be resolved soon.

2. monitoring Apr 02, 2025, 04:32 PM UTC

   The Big Query service latencies have dropped and we see improvements with our internal dashboards.

3. resolved Apr 02, 2025, 05:25 PM UTC

   All of internal alerts are cleared. You should not see delays/errors with dashboards and reports any longer.

Read the full incident report →

Started Mar 21, 2025, 07:57 PM UTC· resolved Mar 21, 2025, 09:13 PM UTC

1. investigating Mar 21, 2025, 08:30 PM UTC

   We are currently investigating this issue.

2. monitoring Mar 21, 2025, 08:44 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 21, 2025, 09:13 PM UTC

   This incident has been resolved.

4. postmortem Mar 28, 2025, 03:14 PM UTC

   **Twingate Public RCA: March 21, 2025 Authentication/Authorization Incident** **Summary** On March 21, 2025, between 21:10 and 21:41 UTC, a subset of authentication and authorization requests to Twingate services experienced elevated error rates. The issue was mitigated by 21:41 UTC and services returned to normal operation. **What Happened** Twingate runs an active-active architecture across multiple Google Kubernetes Engine \(GKE\) clusters, balanced by a Global Load Balancer \(GLB\). At the onset of the incident, we observed anomalies in one of our clusters, and to protect overall system health, we proactively scaled down deployments in that cluster. This action shifted traffic to other clusters in the topology, primarily one that typically handles a lighter load and had been scaled accordingly. The target cluster began autoscaling as expected, but the increased traffic caused elevated error rates, which triggered our retry mechanisms across services. While these retries helped many requests succeed, they also increased the overall system load. Simultaneously, the cluster underwent a cloud provider-initiated update operation that caused pod restarts and reduced capacity. To stabilize the system, we reintroduced capacity in the previously affected cluster, rebalancing the traffic across regions. Once this occurred, error rates subsided and retries diminished. **Root Cause** Anomalous behavior from our cloud provider, including unexpected request timeouts at the load balancer level and instability during a cluster update, led to a cascade of retry traffic that temporarily overwhelmed parts of the system. We are actively investigating both the unexpected timeout configuration and the behavior of the cluster during the update with our cloud provider. **Corrective actions** Short-Term: * Continue collaboration with our cloud provider to understand the root cause of the unexpected timeouts and cluster update impact. * COMPLETED: Keep deployments on all regions with same number of replicas in HPA configuration * COMPLETED: Increase max node counts on cluster node auto-scalers to give us more room to scale up. * COMPLETED: Configure HPA to scale up faster

Read the full incident report →

Started Feb 24, 2025, 08:00 AM UTC· resolved Feb 25, 2025, 01:39 AM UTC

1. investigating Feb 25, 2025, 01:12 AM UTC

   We're currently experiencing a temporary disruption with the DNS Activity Reports. The team is working on restoring access, and we'll have them back up shortly.

2. monitoring Feb 25, 2025, 01:31 AM UTC

   Recent DNS Activity is now restored and we are monitoring the system.

3. resolved Feb 25, 2025, 01:39 AM UTC

   This incident has been resolved.

Read the full incident report →