Mindtickle Outage History

Started May 22, 2025, 06:10 AM UTC· resolved May 22, 2025, 06:44 AM UTC

1. investigating May 22, 2025, 06:10 AM UTC

   Our team is actively investigating the issue and working to restore full access as quickly as possible.

2. identified May 22, 2025, 06:34 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 22, 2025, 06:44 AM UTC

   This incident has been resolved, we will publish a detail root cause analysis soon.

Read the full incident report →

Started May 05, 2025, 08:00 PM UTC· resolved May 05, 2025, 08:00 PM UTC

1. resolved May 06, 2025, 01:51 AM UTC

   The incident occurred on 05 May 2025, between 13:00 - 13:40 PT. This has now been resolved and the platform is working as expected.

2. postmortem May 06, 2025, 05:21 PM UTC

   ### **Impact** On **May 5, 2025**, between **13:00 – 13:40 PT**, users were unable to access the Mindtickle platform, including both the **admin** and **learning** sites. ### **What Happened** * The incident was triggered by a hardware failure within one of our five production database clusters. Specifically, two out of the three database nodes in the affected cluster were simultaneously marked as unhealthy by AWS due to a hardware-level issue. AWS terminated these database nodes, citing "hardware failure." * Unfortunately, we did not find any prior notification regarding the degradation of the hardware for the specific database cluster. This meant that we were unable to proactively respond or mitigate the impact ahead of time. * Once the two database nodes went down, all incoming database requests were redirected to the only remaining database node in the cluster. This database node, having less provisioned capacity, quickly became overwhelmed by the sudden surge in traffic. This led to longer processing queues and, ultimately, request timeouts across the platform. * While our system is designed to tolerate a single database node failure without performance degradation, a simultaneous second database node failure led to service disruption, due to capacity constraints on the third database node. ### **Immediate Actions Taken** * Upon receiving system alerts, our engineering team swiftly identified the root cause within the impacted database cluster. * We initiated a recovery process by provisioning and adding new database nodes back into the cluster. * This remedial action was completed by 14:10 PT, restoring full service availability and normal processing behavior. ### **Learnings and Next Steps** 1. **Remediation for future retirement notices -** We are working with AWS to establish **real-time visibility into instance health and retirement notices via a live dashboard**, instead of relying solely on email notifications. This will allow us to detect and act on early signs of hardware degradation more quickly. 2. **Automated resilience enhancements -** We are actively exploring automation to dynamically add new database nodes with rebalancing when cluster capacity drops below a critical threshold \(e.g., when 2 of 3 database nodes fail\), to maintain performance and availability. We sincerely apologize for the inconvenience caused by this incident. We are committed to taking the necessary steps to enhance our fault tolerance, improve incident response, and strengthen partnerships with our infrastructure providers to avoid such occurrences in the future.

Read the full incident report →

Started Mar 06, 2025, 05:26 AM UTC· resolved Mar 06, 2025, 05:26 AM UTC

1. resolved Mar 07, 2025, 06:38 PM UTC

   Between March 05, 21:26 PT, and March 06, 20:17 PT, we encountered an issue that prevented Zoom call recordings due to an unexpected change in Zoom permissions affecting the Mindtickle integration. We have fully resolved the issue, and your Zoom calls are now being recorded as expected. We sincerely apologize for any disruption this may have caused and are implementing additional safeguards to prevent similar occurrences in the future. A detailed root cause analysis (RCA) will be shared on this status page next week.

Read the full incident report →

Started Jan 21, 2025, 06:44 AM UTC· resolved Jan 21, 2025, 06:58 AM UTC

1. investigating Jan 21, 2025, 06:44 AM UTC

   We are currently investigating this issue.

2. resolved Jan 21, 2025, 06:58 AM UTC

   Mindtickle platform is up and running. A root cause analysis will be published shortly.

Read the full incident report →

Started Jan 10, 2025, 07:47 AM UTC· resolved Jan 10, 2025, 09:41 AM UTC

1. investigating Jan 10, 2025, 07:47 AM UTC

   We are observing high error rate in below workflows 1. JIT search 2. Call QnA 3. AI mission creation 4. AI assessment creation 5. AI mission review 6. Asset email draft generation 7. Guided program creation (GPC) This initial investigation suggests this is linked to the outage on Azure. We are working with the Microsoft team to debug this further.

2. monitoring Jan 10, 2025, 09:07 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 10, 2025, 09:41 AM UTC

   A fix has been implemented and the issue is now resolved. All AI powered workflows are now operational.

4. postmortem Jan 16, 2025, 09:39 AM UTC

   **Incident Summary:** On January 9, 2025, at 11:47 PM PST, AI-powered features of the Mindtickle platform began experiencing a high error rate due to an infrastructure outage at the Microsoft Azure data center. This incident was triggered by a networking configuration issue in the Azure PubSub service. **Impact:** The following workflows were affected by high error rates during the incident: 1. JIT search 2. Call QnA 3. AI mission creation 4. AI assessment creation 5. AI mission review 6. Asset email draft generation 7. Guided program creation \(GPC\) **Timeline \(PST\):** * **\[09-Jan-2025, 11:32 PM PST\]:** Mindtickle systems impacted by Azure Open API outage. * **\[09-Jan-2025, 11:49 PM PST\]:** Alerts received for an increased error rate. * **\[10-Jan-2025, 12:10 AM PST\]:** Root Cause Analysis \(RCA\) identified as a networking configuration issue at Azure. * **\[10-Jan-2025, 12:15 AM PST\]:** Engaged with Azure Support team, to identify possible solutions. * **\[10-Jan-2025, 12:40 AM PST\]:** Fallback deployment initiated in a different Azure availability zone. * **\[10-Jan-2025, 01:41 AM PST\]:** Error rates returned to normal; all systems green. **Resolution:** To mitigate the issue, we redeployed affected services in an alternate Azure availability zone and redirected traffic to these new deployments. This resolved connectivity issues and restored full functionality. **Next Steps:** 1. Collaborate with Azure to ensure the robustness of zonally redundant services and minimize dependency on a single zone. 2. Review and optimize our fallback mechanisms to ensure faster mitigations. 3. Conduct internal drills to simulate similar outages and test disaster recovery strategies. We appreciate your patience and understanding as we work to improve our platform's resilience and minimize the impact of external outages.

Read the full incident report →

Started Jan 02, 2025, 03:20 AM UTC· resolved Jan 02, 2025, 04:50 PM UTC

1. resolved Jan 15, 2025, 01:05 PM UTC

   On January 2, 2025, between 8:50 AM PST to 01:15 PM PST, one of the nodes in our database cluster became unhealthy, causing an increased load on the remaining nodes. This led to sequential unlocking requests getting stuck in a queue and being processed with a delay.

2. postmortem Jan 15, 2025, 01:05 PM UTC

   **Incident Summary:** On January 2, 2025, at 8:50 AM PST, one of the nodes in our database cluster became unhealthy, causing an increased load on the remaining nodes. This led to sequential unlocking requests getting stuck in a queue and being processed with a delay. **Impact:** Users experienced delays in sequential unlocking operations for approximately 4 hours and 25 minutes. No data loss or corruption was observed, but service performance was degraded during this period. **Timeline:** * \[02-Jan-2025, 08:50 AM PST\]: Issue detected with one of the database cluster nodes becoming unhealthy. * \[02-Jan-2025, 09:20 AM PST\]: Sequential unlocking requests began experiencing delays. * \[02-Jan-2025, 01:15 PM PST\]: The unhealthy node issue was resolved, and the backlog of queries began processing. * \[02-Jan-2025, 01:15 PM PST\]: Backlog fully cleared, and normal operations resumed. **Resolution:** The unhealthy node was identified and restored to a healthy state, allowing the system to process the backlog of delayed queries. Once the backlog was cleared, sequential unlocking operations returned to normal functionality. **Next Steps:** To prevent recurrence, the following actions will be taken: 1. Implement additional monitoring and alerting mechanisms to detect similar issues early. 2. Review and optimize our handling of queued requests to minimize delays during high-load scenarios. We sincerely apologize for the inconvenience caused and appreciate your understanding as we work to improve the resilience of our systems.

Read the full incident report →

Started Dec 10, 2024, 03:06 PM UTC· resolved Dec 10, 2024, 03:44 PM UTC

1. investigating Dec 10, 2024, 03:06 PM UTC

   Since 05:38 PT, Dec 10, 2024, we have observed failures for bulk operations and invitation workflows. Below are the workflows that are impacted. 1. Bulk Publish module 2. Bulk archive of module 3. Bulk mirror module 4. Module relevance for a module 5. Module & series Invitation

2. resolved Dec 10, 2024, 03:44 PM UTC

   The incident has been resolved and the system is now back to normal.

3. postmortem Dec 19, 2024, 09:36 AM UTC

   **Incident Summary** On December 10, 2024, an issue was observed where users experienced interruptions when attempting to add, deactivate, or invite users to series and modules. The root cause was traced to a periodic database cleanup activity that took longer than expected, leading to a lag in processing and subsequent errors in the workflow. The issue was promptly identified, and mitigation steps, including stopping the cleanup activity, were executed to restore normal operations. **Impact Area** The following functionalities were impacted during the incident: * Bulk Publish Module * Bulk Archive of Module * Bulk Mirror Module * Update Availability for Module * Module Move * Certification Award * Invitation **Incident Timeline** * **December 10, 2024, 6:06 AM PT**: Users began experiencing issues with workflow functionalities. * **December 10, 2024, 6:20 AM PT**: The first report of the issue was logged. * **December 10, 2024, 6:30 AM PT**: The team initiated an investigation into the root cause. * **December 10, 2024, 6:50 AM PT**: The issue was identified as related to Database cleanup activity. * **December 10, 2024, 7:38 AM PT**: Database cleanup halted, and normal functionality was restored. * **December 10, 2024, 7:44 AM PT**: Issue resolved. **Root Cause Analysis** The issue stemmed from a periodic database cleanup activity that exceeded its expected duration, causing processing delays and errors in key workflows. **Next Steps and Preventive Actions** * **Enhanced Monitoring**: Improved tracking of database cleanup activities to detect and mitigate delays proactively. * **Optimized Cleanup Processes**: Review and optimize database cleanup activities to minimize processing time and ensure stability. * **Improved Workflow Resilience**: Introduce mechanisms to handle delays in dependent processes gracefully without causing errors. We apologize for the inconvenience caused by this incident.

Read the full incident report →

Started Dec 04, 2024, 12:00 PM UTC· resolved Dec 04, 2024, 12:00 PM UTC

1. resolved Dec 20, 2024, 07:10 AM UTC

   On December 4, 2024, we observed issues in the production environment where users experienced errors on authentication workflows, including signup, user impersonation, and forgot-password functionalities.

2. postmortem Dec 20, 2024, 07:11 AM UTC

   **Incident Summary** On December 4, 2024, we observed issues in the production environment where users experienced errors on authentication workflows, including signup, user impersonation, and forgot-password functionalities. The issue stemmed from a database performance problem that impacted multiple operations reliant on the same table. Upon investigation, The root cause was identified as an unexpected surge in load on the mobile endpoint, leading to database performance issues. While this caused disruptions to various workflows, immediate mitigation efforts were undertaken to restore services. **Impact Area** * **Signup Flows**: Errors in both mobile and web-based signup workflows. * **User Impersonation**: Errors experienced by specific customers. * **Forgot-Password Flows**: Delays in password reset functionality. **Incident Timeline** * **December 4, 2024, 4:00 AM PT**: An increase in requests to the signup endpoint was detected, accompanied by errors. * **December 4, 2024, 4:01 AM PT**: Alerts triggered regarding elevated errors in the login-provider service. * **December 4, 2024, 4:25 AM PT**: Root cause identified as table locks due to surge in requests. * **December 4, 2024, 4:35 AM PT**: Decision made to purge outdated records from the affected table to reduce load. * **December 4, 2024, 4:55 AM PT**: Purging of old records initiated. * **December 4, 2024, 5:10 AM PT**: Purging completed successfully, clearing approximately 2 million old records. * **December 4, 2024, 5:15 AM PT**: Services restored, and all impacted workflows resumed normal operation. **Root Cause Analysis** The root cause of the incident was an unexpected surge in traffic on the /mobile/signup endpoint_._ The increased load led to performance bottlenecks in the database, particularly with queries handling old records. **Next Steps and Preventive Actions** * **Load Testing**: Enhance load testing processes to include scenarios with 10x normal traffic levels. * **Traffic Monitoring**: Set up advanced monitoring for sudden traffic spikes and introduce rate-limiting mechanisms if necessary. * **Query Performance**: Optimize database queries to prevent bottlenecks during peak traffic. We sincerely apologize for any inconvenience caused during this time and appreciate your patience. Should you have further questions or concerns, please reach out to our support team.

Read the full incident report →

Started Nov 27, 2024, 06:06 PM UTC· resolved Nov 27, 2024, 11:46 PM UTC

1. investigating Nov 27, 2024, 06:06 PM UTC

   Since 07:06 PT, Nov 27, 2024, we have observed intermittent failures for some bulk operations and invitation workflows. Below are the flows that are impacted. 1. Module-related bulk operations from series page [copy, publish, move, mirror] 2. Invitation: manual & invite during publish / mirror 3. Certification: delay in delivery of system-based certificates 4. Bulk Uploads The team is investigating the issue and we will share an update shortly.

2. identified Nov 27, 2024, 11:45 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Nov 27, 2024, 11:46 PM UTC

   The incident has been resolved and the system is now back to normal.

4. postmortem Dec 05, 2024, 09:14 AM UTC

   ### **Incident Summary** On **Nov 27, 2024**, we experienced a large influx of **requests** through the Open API. The requests were bulk API calls, amounting to a **few million new requests.** Each request was sent to a queue for efficient and reliable processing. Due to erroneous routing configuration, these requests were also sent to a dormant queue that had no consumer - leading to the queue getting full and memory overload in the message queue system. At **8:43 pm PT**, the team began receiving alerts. This was traced back to a **memory overload** in the message queue system caused by a large volume of unprocessed events. This incident led to intermittent failures in several workflows, including **user sync, invitations, certifications, and bulk uploads**. #### **Impacted Workflows:** * Bulk Publish module * Bulk Archive module * Bulk Mirror module * Update Availability module * Module Move * Certification Award * Schedule Invitation * Bulk update through Open APIs * User Sync ### **Incident Timeline** * **Nov 27, 2024, 7:06 AM PT**: Workflow service began encountering errors, marking the start of the incident. * **Nov 27, 2024, 8:43 AM PT**: War room activated to address the issue. * **Nov 27, 2024, 8:54 AM PT**: Identified the root cause of errors was the messaging queue system’s memory overload. * **Nov 28, 2024, 10:00 AM PT**: Determined that system upgrades / upscale could not occur during high memory usage. * **Nov 28, 2024, 11:15 AM PT**: Determined high memory overload was due to erroneous unnecessary routing of events to a queue that had no consumer leading to the queue getting full. Decision made to purge the faulty queue to reduce memory usage. * **Nov 28, 2024, 11:30 AM PT**: Purge completed and monitored the system for memory reduction. * **Nov 28, 2024, 12:30 PM PT**: Memory usage remained high despite the purge. * **Nov 28, 2024, 1:04 PM PT**: Performed a force reboot of the system, and memory usage normalized. * **Nov 28, 2024, 3:36 PM PT**: Systems returned to normal. ### **Root Cause Analysis** The incident was caused by a **memory overload** in the message queue system, triggered by the overflow of unprocessed events. This overload prevented workflows from completing, impacting several services. ### **Lessons Learned** * **Message Queue Configuration Automation**: The absence of automated synchronization between code changes and message queue configurations led to missing routing keys, causing issues with the DLQ. A system to automatically update message queue configurations will help mitigate this risk. * **Unmonitored DLQs**: This dead-letter queue \(DLQs\) was not actively monitored, which led to an overflow of unprocessed events. Future processes should include dedicated monitoring for DLQs, along with a consumer to process stalled events. * **Delayed Detection and Resolution**: The incident took too long to detect and resolve. By implementing **improved monitoring**, **better-alerting systems**, and **real-time anomaly detection**, we can reduce **MTTR** \(Mean Time to Resolution\) and **MTTD** \(Mean Time to Detection\) for future incidents.

Read the full incident report →