TeamDynamix had 35 outages in the last 2 years totaling 513h 32m of downtime — averaging 1.4 incidents per month.
There were 35 TeamDynamix outages since May 30, 2024 totaling 513h 32m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are investigating a delay in webhooks executions that began approximately around 2:14 PM ET. We're taking steps to mitigate the delays as soon as possible.
As of 4:19 PM EST, webhooks are caught back up and processing as expected.
TeamDynamix is currently investigating reports of latency and increased error rates. We will provide updates as soon as possible, though database reporting metrics appear to have returned to normal as of this time.
After monitoring to confirm, performance has returned to normal as of last night. The issue was caused by higher than normal traffic levels for client portal resources.
We are currently investigating an issue where email monitors are not draining as expected.
IMAP email monitors were not processing in the East US environment from 12:00-1:45 ET. This issue is now resolved and email monitors are processing as expected.
We are currently investigating an issue that caused iPaaS to be unresponsive between 12:37PM ET to 12:48PM ET.
We are currently investigating an issue that caused some Work Management pages and iPaaS to be unresponsive between 12:37PM ET to 12:48PM ET.
The U.S. tenant of iPaaS/CAI had a brief moment of UI unavailability, and flows responding with 502 bad gateway error codes between 12:37PM ET and 12:48PM ET. This issue is now resolved.
We are currently investigating reports of performance degradation around the iPaaS user interface page loads. At this time iPaaS flows are executing at expected levels and remain fully operational.
Shortly after this issue arose we did see performance levels return quickly to expected metrics. At this time we will continue monitoring as the root cause is diagnosed, but the web interface should be completely operational based on present measurements.
A high-memory usage scenario caused a brief period of slowness within the UI. We were able to isolate the instance and restart which resolved the UI slowness by 11am EST yesterday. We are currently evaluating options to prevent this behavior in the future.
We are currently investigating reports of unhandled errors when accessing tickets in Work Management.
We’ve deployed a fix to address unhandled errors when accessing tickets in Work Management. The issue was caused by a misconfiguration during a maintenance deployment and has now been fully resolved.
We have noticed an increased error rate in the loading of charts and are investigating.
We have released a hotfix that resolved the bug that caused this issue.
We have noticed instances of increased response times and are taking corrective efforts to reduce those to normal levels. We'll provide updates here as they are available.
Hotfixes have been implemented to address elevated response times caused by processing the backend data for dashboard widgets. We have observed performance returned to normal levels as of about 11:20 AM ET.
We are currently investigating and taking steps to address reports of increased response times in WorkManagement, mainly centered around dashboard interaction/load times.
We have resolved the dashboard-related instances of response time elevation issues.
The 12.1 production upgrade is live for the majority of the system. We are still processing a cutover which impacts the following services: Item Alerts, KB Article Review Reminders, Project Health Updates, Report Delivery, RSS Feeds, Source Control Events, Ticket SLAs and Ticket Workflow Web Service Steps. We will continue to provide updates as the cutover progresses.
The 12.1 cutover is complete and backend services are now running again.
Beginning at around 11:30AM TeamDynamix began to experience latency with our Support ticket outbound email notifications due to the ongoing Amazon AWS outages. This issue is delaying the delivery of outbound notifications when customers submit support tickets or when TeamDynamix personnel respond to items from within our environment which should relay a notification to any customer via emial. We'll provide an update as soon as the AWS outage has been confirmed as resolved or when we see the message delivery delay resolved.
We are continuing to investigate this issue.
At this time we have successfully processed through all backlogged outbound notifications that had queued for delivery. We will continue to monitor the AWS incident until they resolve all remaining latency issues.
AWS services returned to normal for us by around 3:30 PM ET yesterday. This issue resolved as of that time.
We are experiencing a longer than normal level of webhook event processing times in Sandbox, due to an excessive backlog of webhook requests. The system is chewing through the backlog of queued events, and we will scale as appropriate until everything is caught up again.
Webhook backlogged event requests are fully caught up again in Sandbox.
We are currently investigating an iPaaS issue where a limited number of iPaaS flows are failing with the error message: "No Indexer available to resolve Identifier." This issue is impacting only a small subset of flows in isolated cases. Our engineering team is actively working to identify the root cause and will provide updates as more information becomes available.
We have identified the source of the issue and are developing a fix.
The problem has been patched in Production and was confirmed to have resolved the issue.
We are actively investigating an issue around users being unable to either create or edit reports.
A hotfix was deployed around or shortly after 11 PM ET which resolved the issue in both Sandbox and Production.
We have received reports about, and are actively investigating, an issue with email monitors failing to process mail after this weekend's maintenance period.
We are continuing to investigate this issue.
This appeared to have only impacted monitors still using IMAP type auth accounts. We successfully restarted an application that did not come back online as expected during the maintenance, which has resolved the issue.
TeamDynamix is investigating reports of "unable to load email message" errors. Investigation is ongoing to determine a cause and address.
TeamDynamix identified an issue loading up emails which led to a backlog of requests, causing slowness and errors in email monitoring. We are investigating additional safeguards to prevent this from happening in the future and will be implementing preventative measures accordingly.
On 5/14 from about 1:14pm – 1:23pm ET TeamDynamix experienced a brief, but isolated period of performance degradation. This degradation generally presented in slowness while interacting with client portal functions. The issue quickly resolved itself, and we are actively investigating potential expansion of infrastructure controls to prevent this from taking place again.
We are investigating the source of the performance degradation that began a short time ago. As of now it seems to have been a brief spike that has subsided. We'll continue to monitor and determine the root cause.
After monitoring things for a time we have confirmed this issue as resolved.
We are investigating reports of iPaaS loading slowly, or throwing 500 errors on pages trying to load. At the moment, our U.S. customers may experience degraded performance interacting with the iPaaS system.
At 2:30PM EST, the iPaaS & Conversational AI platform experienced an unexpected service disruption lasting until roughly 4:00PM EST. We identified the issue as a series of compounding process executions which ultimately led to degraded performance for our U.S. clients. We have since isolated, and contained the behavior returning system performance back to normal. Our team is still actively evaluating safeguard options to prevent this behavior from occurring going forward.
US iPaaS briefly went offline this morning for a few minutes. Systems were quickly returned to operability, and we are actively monitoring to ensure stability remains consisten.
A misconfiguration during a planned maintenance update of iPaaS resulted in bringing it offline briefly. The iPaaS team immediately corrected the configuration which brought services back online within a space of just a few minutes. At this time the incident is considered resolved.
We are currently investigating intermittent errors being shown when accessing production.
This issue was determined to be the result of a deploy issue from this weekend’s maintenance deploy, but has now been resolved.