Admin console billing functionality is temporarily unavailable

Update timeline

1. investigating Dec 15, 2021, 07:12 PM UTC

   The admin console is currently experiencing an outage caused by our 3rd party billing system being unavailable. Our team is investigating the issue and working on a fix currently. This issue is isolated to loading the admin console and does not affect any resource access or end user authentication.

2. identified Dec 15, 2021, 07:13 PM UTC

   The root cause of the issue has been identified and a fix is in progress. We will post another update shortly.

3. monitoring Dec 15, 2021, 07:33 PM UTC

   A fix has been deployed and the admin console is now available to all customers. We'll continue to monitor the system for the next 30 minutes before marking this incident as resolved.

4. monitoring Dec 15, 2021, 08:05 PM UTC

   We have received confirmation from our 3rd party billing provider that they are aware of their system outage and are working on a fix. We will resolve this incident when the 3rd party system is available. Until this incident is marked as resolved, no billing functionality will be available, but all other functionality remains unaffected.

5. resolved Dec 15, 2021, 09:22 PM UTC

   Our 3rd party billing system is now available and we are marking this incident as resolved. We will follow up with a post mortem update.

6. postmortem Dec 15, 2021, 09:27 PM UTC

   **Components impacted** * Admin console **Summary** At 18:59 UTC we received an automated alert that requests to our 3rd party billing system \(Chargebee\) were failing. A few minutes later we confirmed that this was causing the Twingate admin console interface to fail to load with a 500 error, impacting all customers. At 19:15 UTC our engineering team submitted a hotfix to resolve the issue and update our production systems. The fix was deployed at 19:33 UTC and all customers were able to access the Twingate admin console with billing functionality disabled. We spoke to Chargebee at 20:00 UTC and they confirmed the issue on their side. At 20:47 UTC Chargebee confirmed that the outage was resolved in their system and the incident was closed. **Root cause** The Twingate admin console relies on Chargebee API access in order to load billing information specific to the particular customer account. This API call is made when the Twingate admin console loads in the browser. The underlying 3rd party API returned a 503 \(Service Temporarily Unavailable\) error, which was not captured as an exception in the error object returned by the 3rd party API library. This led to an uncaught exception, which caused the admin console to fail to load with a generic 500 \(Internal Server Error\) error. **Corrective actions** We have updated the admin console logic to capture this type of exception from Chargebee and ensure that the admin console will continue to load. We will be taking the following actions: 1. Auditing all 3rd party calls in the admin console to ensure that all exceptions are caught and do not result in the admin console being unavailable. 2. Update the billing behavior specifically to incorporate an unavailability message.