Tenfold experienced a major incident on July 17, 2018 affecting API and Dashboard, lasting 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 17, 2018, 07:29 PM UTC
Our automated systems have alerted the Tenfold site reliability team of elevated error rates with the Dashboard UI and API. The team is currently investigating the cause of the behavior, and more information will be provided as it is made available.
- monitoring Jul 17, 2018, 07:46 PM UTC
Tenfold site reliability team has identified root cause of Dashboard and API errors to be related to an ongoing incident with Google Cloud Global Loadbalancing, outlined here: https://status.cloud.google.com/incident/cloud-networking/18012 Automated failover methods have taken over and system is normalizing. Tenfold team will be closely monitoring performance during impacted period with Google Cloud and adding additional resources, if necessary.
- resolved Jul 17, 2018, 08:29 PM UTC
The issue with Google Cloud Global Load balancers returning 502s has been resolved for all affected users as of 13:05 US/Pacific. Additional details can be found in their Status page here: https://status.cloud.google.com/incident/cloud-networking/18012 Should you have any additional questions, or concerns, regarding Tenfold functionality feel free to reach out to our support team: e: [email protected]