Twingate experienced a minor incident on July 13, 2021 affecting Americas Relays, lasting 11h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 13, 2021, 04:58 PM UTC
We have identified an issue with our Relay cluster in Mumbai (GCP region asia-south1-a) and are working to fix it.
- monitoring Jul 13, 2021, 06:31 PM UTC
Engineering has a fix in place. We are currently monitoring the cluster and expect it to be back up by 21:00 PST.
- resolved Jul 14, 2021, 03:58 AM UTC
Relay cluster in Mumbai (GCP region asia-south1-a) is now fully operational.
- postmortem Jul 29, 2021, 06:33 PM UTC
**Components impacted** Relay **Summary** We encountered an issue during an upgrade to our relay cluster monitoring infrastructure. As a result, we were unable to bring the Mumbai regional cluster up during this maintenance window, and so it was left down. There was no customer impact as any connections were re-routed to another relay cluster. **Root cause** We determined that the root cause was a configuration error introduced during a deployment configuration upgrade. This was fixed and the cluster was brought back up during a low traffic period at the end of the day. **Corrective actions** We identified an issue in our CI/CD process that resulted in the initial misconfiguration, which has been corrected.