TeamDynamix Outage History
TeamDynamix degraded · 1 active incident View live status →Pingoru has recorded 52 incidents for TeamDynamix: 24 in the last 12 months and 6 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 2 | 73 of 744 (10%) |
| June 2026 | 2 | 20 of 720 (3%) |
| July 2026 | 2 | 3 of 744 (0%) |
| August 2026 | 1 | 2 of 744 (0%) |
| September 2026 | 2 | 12 of 720 (2%) |
| October 2026 | 2 | 204 of 573 (36%) |
Incidents, newest first
Minor July 14, 2025 Investigating Work Management issues around report creation and editing 18h 1m Web Interface
Started Jul 14, 2025, 06:47 PM UTC· resolved Jul 15, 2025, 12:48 PM UTC
- investigating Jul 14, 2025, 06:47 PM UTC
We are actively investigating an issue around users being unable to either create or edit reports.
- resolved Jul 15, 2025, 12:48 PM UTC
A hotfix was deployed around or shortly after 11 PM ET which resolved the issue in both Sandbox and Production.
Major June 28, 2025 Investigating report of email monitor errors/not processing emails 41m Email, Email
Started Jun 28, 2025, 08:36 PM UTC· resolved Jun 28, 2025, 09:18 PM UTC
- investigating Jun 28, 2025, 08:36 PM UTC
We have received reports about, and are actively investigating, an issue with email monitors failing to process mail after this weekend's maintenance period.
- investigating Jun 28, 2025, 08:59 PM UTC
We are continuing to investigate this issue.
- resolved Jun 28, 2025, 09:18 PM UTC
This appeared to have only impacted monitors still using IMAP type auth accounts. We successfully restarted an application that did not come back online as expected during the maintenance, which has resolved the issue.
Minor May 28, 2025 Investigating Reports of Email Monitor Errors 15h 29m Email
Started May 28, 2025, 09:54 PM UTC· resolved May 29, 2025, 01:24 PM UTC
- investigating May 28, 2025, 09:54 PM UTC
TeamDynamix is investigating reports of "unable to load email message" errors. Investigation is ongoing to determine a cause and address.
- resolved May 29, 2025, 01:24 PM UTC
TeamDynamix identified an issue loading up emails which led to a backlog of requests, causing slowness and errors in email monitoring. We are investigating additional safeguards to prevent this from happening in the future and will be implementing preventative measures accordingly.
Notice May 15, 2025 Performance degradation on 5/14 –
Started May 15, 2025, 02:01 PM UTC· resolved May 15, 2025, 02:01 PM UTC
- resolved May 15, 2025, 02:01 PM UTC
On 5/14 from about 1:14pm – 1:23pm ET TeamDynamix experienced a brief, but isolated period of performance degradation. This degradation generally presented in slowness while interacting with client portal functions. The issue quickly resolved itself, and we are actively investigating potential expansion of infrastructure controls to prevent this from taking place again.
Notice April 28, 2025 Investigating Performance spikes beginning around 11 AM ET 1h 48m Web Interface
Started Apr 28, 2025, 03:11 PM UTC· resolved Apr 28, 2025, 05:00 PM UTC
- monitoring Apr 28, 2025, 03:11 PM UTC
We are investigating the source of the performance degradation that began a short time ago. As of now it seems to have been a brief spike that has subsided. We'll continue to monitor and determine the root cause.
- resolved Apr 28, 2025, 05:00 PM UTC
After monitoring things for a time we have confirmed this issue as resolved.
Minor April 22, 2025 Investigating iPaaS and Conversational AI performance degradation 2h Flow Execution, Web Interface
Started Apr 22, 2025, 06:46 PM UTC· resolved Apr 22, 2025, 08:47 PM UTC
- investigating Apr 22, 2025, 06:46 PM UTC
We are investigating reports of iPaaS loading slowly, or throwing 500 errors on pages trying to load. At the moment, our U.S. customers may experience degraded performance interacting with the iPaaS system.
- resolved Apr 22, 2025, 08:47 PM UTC
At 2:30PM EST, the iPaaS & Conversational AI platform experienced an unexpected service disruption lasting until roughly 4:00PM EST. We identified the issue as a series of compounding process executions which ultimately led to degraded performance for our U.S. clients. We have since isolated, and contained the behavior returning system performance back to normal. Our team is still actively evaluating safeguard options to prevent this behavior from occurring going forward.
Minor April 9, 2025 Conversational AI: degraded performance 7h 6m Flow Execution
Started Apr 09, 2025, 01:01 PM UTC· resolved Apr 09, 2025, 08:08 PM UTC
- investigating Apr 09, 2025, 01:01 PM UTC
We have noticed Azure OpenAI Service and Azure AI Services currently show elevated latency in East US since 11:20pm EST on 4/8. MS indicates a fix is in process but no estimated time to fix. We're only seeing this impact *some* Conversational AI customers, but for any impacted, your bots MAY see degradation around usage of AI Knowledge Sets, bot translations, and processes utilizing the “Generate” step. Knowledge set jobs are also impacted.
- identified Apr 09, 2025, 03:51 PM UTC
The underlying issue is still in the process of being fixed by Microsoft, however we've noticed also that bot training & intent matching may be potential impacted areas.
- identified Apr 09, 2025, 04:07 PM UTC
The following communication was received from Microsoft this morning regarding an ongoing Conversational AI bot behavior issue that TDX has recognized: ~~~~~ Impact Statement: Starting at 02:30 UTC on 09 April 2025, you/TeamDynamix were identified as a customer using Azure OpenAI Service and Azure AI Services in East US, who may experience elevated latencies and availability issues across all AOAI models. Current Status: We determined that a recent deployment impacted backend instances upon which Azure OpenAI Service and Azure AI Services rely, which is leading to latency and availability problems. Our mitigation strategy consisted of performing a rollback to a previous, known build; however, it has not proven effective at alleviating impact. We are now focusing our efforts on identifying an appropriate mitigation path to resolve this issue. The next update will be provided in 2 hours or as events warrants. ~~~~~ Please note the Azure status page shows no issues unless signed in as an Azure Service User. TeamDynamix will continue to monitor their progress and will advise as soon as they indicate success in implementing a fix or other mitigating measures. At this time, the behaviors noted previously are being caused by this Azure OpenAI and AI Services issue (Azure tracking ID 1T3L-3QG). To this point, we've noted issues with usage of AI Knowledge Sets, bot translations, and processes utilizing the “Generate” step, Knowledge set jobs, bot training & intent matching. This is only affecting some Conversational AI customers/environments from our best assessment.
- monitoring Apr 09, 2025, 05:29 PM UTC
Azure has notified us that this issue has been resolved through mitigating steps. As we receive that specific resolution information from them, we will provide it here on this incident.
- resolved Apr 09, 2025, 08:08 PM UTC
Closing as the issue has been confirmed as resolved. Further details will be added when/if we receive them from Azure on the issue.
Notice January 29, 2025 US iPaaS inaccessible 14m Web Interface
Started Jan 29, 2025, 02:38 PM UTC· resolved Jan 29, 2025, 02:53 PM UTC
- monitoring Jan 29, 2025, 02:38 PM UTC
US iPaaS briefly went offline this morning for a few minutes. Systems were quickly returned to operability, and we are actively monitoring to ensure stability remains consisten.
- resolved Jan 29, 2025, 02:53 PM UTC
A misconfiguration during a planned maintenance update of iPaaS resulted in bringing it offline briefly. The iPaaS team immediately corrected the configuration which brought services back online within a space of just a few minutes. At this time the incident is considered resolved.
Minor January 25, 2025 Intermittent Errors in Production 11h 32m Web Interface, Web Interface
Started Jan 25, 2025, 02:11 AM UTC· resolved Jan 25, 2025, 01:43 PM UTC
- investigating Jan 25, 2025, 02:11 AM UTC
We are currently investigating intermittent errors being shown when accessing production.
- resolved Jan 25, 2025, 01:43 PM UTC
This issue was determined to be the result of a deploy issue from this weekend’s maintenance deploy, but has now been resolved.
Minor December 14, 2024 Experiencing errors when accessing Sandbox 2d 15h Sandbox, Sandbox
Started Dec 14, 2024, 12:25 AM UTC· resolved Dec 16, 2024, 03:56 PM UTC
- investigating Dec 14, 2024, 12:25 AM UTC
We are investigating an issue where some Sandbox environments are presenting the following error: “We're sorry, but there has been an error in this application.”
- investigating Dec 14, 2024, 12:42 AM UTC
We are continuing to investigate this issue.
- resolved Dec 16, 2024, 03:56 PM UTC
This issue was determined to be the result of a deploy issue from this weekend's maintenance deploy. We were able to pinpoint the issue and correct it as of Saturday morning. This is now considered resolved as of the correction from Saturday morning.
Minor November 25, 2024 Experiencing Email Monitor processing delays 19h 57m Email
Started Nov 25, 2024, 06:00 PM UTC· resolved Nov 26, 2024, 01:57 PM UTC
- investigating Nov 25, 2024, 06:00 PM UTC
TeamDynamix has noticed an issue causing delays in emails being processed, and with messages not getting moved to the appropriate destination folder once processed. Investigation is ongoing to determine a cause and address.
- investigating Nov 25, 2024, 06:15 PM UTC
We believe this may be related at least to some degree to Microsoft's ongoing issue which has the ID of: MO941162
- resolved Nov 26, 2024, 01:57 PM UTC
The issues caused by Microsoft incident MO941162 began to resolve at or around 5 PM ET yesterday. Once that cleared up, we saw behavior of email monitoring return to normal shortly thereafter, and have determined the root cause to have been related to that Microsoft systems issue in full. As of yesterday evening, this issue is considered resolved.
Minor November 22, 2024 Paused iPaaS flows and CAI conversations 5m Flow Execution, Flow Execution
Started Nov 22, 2024, 09:10 PM UTC· resolved Nov 22, 2024, 09:16 PM UTC
- investigating Nov 22, 2024, 09:10 PM UTC
iPaaS flows and CAI conversations are paused due to a failed patch attempt the afternoon of 11/22/2024. We are rolling back the patch to resolve and resume normal processing.
- resolved Nov 22, 2024, 09:16 PM UTC
Successfully rolled back to version 3.2.0.37 - flows and conversations should be resuming as expected.