TeamDynamix Outage History

This issue was determined to be the result of a deploy issue from this weekend's maintenance deploy. We were able to pinpoint the issue and correct it as of Saturday morning. This is now considered resolved as of the correction from Saturday morning.

The issues caused by Microsoft incident MO941162 began to resolve at or around 5 PM ET yesterday. Once that cleared up, we saw behavior of email monitoring return to normal shortly thereafter, and have determined the root cause to have been related to that Microsoft systems issue in full. As of yesterday evening, this issue is considered resolved.

Successfully rolled back to version 3.2.0.37 - flows and conversations should be resuming as expected.

Yesterday evening the TeamDynamix email monitoring systems experienced a backup in our back-end infrastructure that required manual intervention to resolve. After intervention completed, the backup proceeded to clear up in a short matter of time. During the period that the backup was in progress, many errors were generated stating an email "couldn't be found or is in the wrong state". These should have ceased to occur as of late last night. The incident is considered already resolved and has been posted here for insight due to its impact.

Yesterday evening TeamDynamix performed maintenance on pieces of our infrastructure. This maintenance unfortunately resulted in the webhook systems not being notified about change events properly. We have corrected this issue this morning and have put measures in place to prevent that state of behavior from being possible going forward. The unfortunate news is the change/create webhook events that happened during the time things were in a bad state are not going to fire (they did not queue up), but webhooks should be firing now as expected and we consider the incident to be resolved.

Infrastructure servers were identified as causing these error messages and were removed, which has resolved the issues as of 9am ET.

This incident has been resolved.

Firewall maintenance was recently completed to prioritize HTTP/2 connections for performance purposes. Certain versions of the Google Chrome browser were having difficulty establishing connection via HTTP/2. In this scenario, browsers should fall back to HTTP/1.1, yet that was not occurring successfully for many users running Google Chrome. Given the popularity of the Google Chrome browser, TeamDynamix has disabled HTTP/2 until further analysis can be performed. This fix is confirmed as successful.

Webhook backlogged event requests were fully caught up again by about 2:45 ET.

This incident has been resolved.