Mindtickle Outage History

Started Sep 21, 2026, 01:13 PM UTC· resolved Sep 21, 2026, 04:42 PM UTC

1. investigating Sep 21, 2026, 01:13 PM UTC

   Users are currently unable to log in to the Mindtickle platform. Users who are already logged in are not impacted and can continue using the platform. Our team is actively investigating the issue.

2. identified Sep 21, 2026, 01:53 PM UTC

   We have identified the root cause of the login issue and are currently working on a patch.

3. monitoring Sep 21, 2026, 03:10 PM UTC

   We are seeing signs of recovery, and some users should now be able to log in to the Mindtickle platform. We are continuing to monitor the recovery and work through the remaining impacted users.

4. monitoring Sep 21, 2026, 03:46 PM UTC

   Update: Login functionality has now been restored for the majority of users. We will continue to monitor the platform closely to ensure full recovery and stability.

5. resolved Sep 21, 2026, 04:42 PM UTC

   The login issue affecting the platform has been resolved, and users can now sign in normally. The disruption lasted from 04:52 AM PT to 9:42 AM PT. No customer data was affected. Users who were already logged in to Mindtickle were able to access the platform without any concerns. We're sorry for the disruption this caused. If you're still unable to log in, please contact us @ [email protected]. We'll complete an internal postmortem and publish the RCA here within the next few days.

Read the full incident report →

Started Sep 16, 2026, 01:33 PM UTC· resolved Sep 16, 2026, 05:01 PM UTC

1. identified Sep 16, 2026, 01:33 PM UTC

   We are aware of an issue affecting Salesforce-related workflows, caused by an ongoing outage on Salesforce's side. This is not the result of an issue within Mindtickle. What's affected: Customers using the Salesforce integration may see failures or delays in the following: 1. Salesforce Integration (SFDC): CRM data sync, user provisioning/sync, module/assets recommendations, and learner performance sync to Salesforce 2. Digital Sales Room: syncing opportunities, contacts, automatic room creation for Salesforce opportunities, and accounts from Salesforce, and writing room activity back to Salesforce. Core DSR functionality (creating and sharing rooms, content, analytics) remains available. 3. Call AI: mapping recordings to Salesforce records and showing Salesforce opportunity/account fields on a meeting once it's available, pushing call insights back to Salesforce, and Salesforce Dialer call capture. Core Call AI functionality (recording, transcription, insights) remains available. What's not affected: Non-Salesforce workflows, core product functionality, and customers not using the Salesforce integration are all operating normally. We are actively monitoring the Salesforce outage and assessing impact.

2. resolved Sep 16, 2026, 05:01 PM UTC

   The Salesforce-side outage has been resolved, and all affected workflows are now operating normally.

Read the full incident report →

Started Aug 31, 2026, 11:08 AM UTC· resolved Aug 31, 2026, 01:06 PM UTC

1. investigating Aug 31, 2026, 11:08 AM UTC

   Seller Copilot is experiencing high error rates due to a downstream outage in the Azure data center supporting this use case.

2. identified Aug 31, 2026, 11:40 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 31, 2026, 12:10 PM UTC

   a fix has been implemented and we are monitoring the situation.

4. resolved Aug 31, 2026, 01:06 PM UTC

   The incident has been resolved, and all systems are now operational.

Read the full incident report →

Started Aug 19, 2026, 05:16 PM UTC· resolved Aug 20, 2026, 02:07 AM UTC

1. investigating Aug 19, 2026, 05:16 PM UTC

   We're investigating an issue where a subset of customers may see stale data in certain analytics reports, specifically: SSR, Custom Dashboards, Manager & Relationship reports, and Overview / Adoption / Performance reports. Other reports and product areas are unaffected. If your analytics reflect current data, no action is needed. We'll post an update shortly.

2. resolved Aug 20, 2026, 02:07 AM UTC

   This issue is resolved. All affected reports are now showing current data. Thank you for your patience.

Read the full incident report →

Started Jun 13, 2026, 12:35 AM UTC· resolved Jun 15, 2026, 03:36 AM UTC

1. investigating Jun 13, 2026, 12:35 AM UTC

   We are currently experiencing a partial service disruption on Mindtickle platform. Users may encounter errors, intermittent failures, or degraded performance across the platform. Our engineering team is actively investigating and mitigating the issue while closely monitoring recovery efforts.

2. identified Jun 13, 2026, 12:51 AM UTC

   We have identified the cause of the elevated error rates and are deploying mitigation measures. We are monitoring the platform closely and working to restore normal service.

3. monitoring Jun 13, 2026, 01:01 AM UTC

   Mitigation has been deployed and services are operating normally. We are continuing to monitor the platform closely to ensure stability.

4. resolved Jun 15, 2026, 03:36 AM UTC

   The incident has been resolved and services are operating normally. We will continue to monitor the platform to ensure stability. A detailed Root Cause Analysis will be shared shortly.

5. postmortem Jun 18, 2026, 12:24 PM UTC

   # **Incident Summary** On June 12, 2026, Mindtickle experienced a service disruption caused by an unexpected surge in processing activity within one of our backend workflows. The resulting increase in downstream traffic exceeded the capacity of a critical platform dependency, leading to service degradation and eventual unavailability. The issue was mitigated by reducing processing throughput and allowing affected systems to recover. **Impact Duration:** **3 hours and 15 minutes**. ## **Impact** During the incident window, customers may have experienced: * Delays in invitation-related workflows * Elevated latency and intermittent errors across affected services * Temporary degradation of platform functionality dependent on the impacted backend systems ## **Incident Timeline** * ‌**Jun 12, 03:00 PM**: A critical backend dependency becomes saturated, resulting in service degradation and elevated error rates. * **Jun 12, 03:15 PM**: The issue is detected through automated monitoring and customer reports. Investigation begins. * **Jun 12, 04:50 PM**: Mitigation measures are implemented to reduce processing throughput and stabilize affected systems. * **Jun 12, 05:46 PM**: Platform functionality is restored fully ## **Root Cause** A backend workflow generated a significantly higher volume of processing activity than historical norms. As the system worked through the resulting backlog, requests propagated through multiple downstream services, creating sustained load across the platform. While each component behaved as designed, the combined volume exposed capacity and protection gaps within the processing chain. The resulting amplification effect overwhelmed a critical backend dependency, causing service degradation and eventual unavailability. ## **Resolution** The team mitigated the incident by reducing the rate at which backlog processing occurred, allowing downstream systems to stabilize and recover. Once traffic levels returned to normal operating ranges, affected services resumed normal operation and platform functionality was fully restored. ## **Corrective and Preventive Actions** * Introduce rate limiting and additional safeguards for high-volume processing workflows to prevent a single workflow from generating excessive downstream load. * Implement backpressure, circuit-breaking, and load-shedding mechanisms across critical processing paths to better protect downstream dependencies during traffic spikes. * Review platform capacity limits, concurrency settings, and monitoring to improve resilience and provide earlier visibility into abnormal processing patterns.

Read the full incident report →

Started Jun 02, 2026, 02:19 PM UTC· resolved Jun 02, 2026, 04:21 PM UTC

1. resolved Jun 03, 2026, 05:06 PM UTC

   On June 2nd, between 7:19 AM PDT and 9:21 AM PDT, our third-party recording provider experienced a disruption that prevented the Call AI bot from joining meetings. A limited number of customers were impacted, and unfortunately, the affected recordings are non-recoverable. We sincerely apologize for the disruption and any impact to your workflows. The issue is fully resolved, and recordings are functioning normally. If your account was affected, your account team will reach out with specific details.

Read the full incident report →

Started May 21, 2026, 03:29 PM UTC· resolved May 21, 2026, 06:30 PM UTC

1. investigating May 21, 2026, 03:29 PM UTC

   Users may experience issues starting Two-Way Roleplay sessions. In addition, completed roleplays may not be accessible, and admins may be unable to create new roleplays. The team is actively investigating the issue and working toward resolution.

2. identified May 21, 2026, 03:59 PM UTC

   We have identified the root cause as a service disruption at a third-party provider that powers parts of the Two-Way Roleplay workflow. We are actively working with the vendor to restore the service and recover impacted functionality as quickly as possible.

3. identified May 21, 2026, 05:07 PM UTC

   Recovery efforts are still ongoing, and we continue to work closely with our third-party partners to restore the impacted service. Our teams remain actively engaged on the issue.

4. identified May 21, 2026, 05:18 PM UTC

   We are seeing partial recovery in the impacted service, and some roleplay attempts are now completing successfully. We continue to work with the third-party provider toward full restoration.

5. monitoring May 21, 2026, 05:49 PM UTC

   The impacted systems are now operational again, and roleplay workflows are functioning successfully. We are continuing to monitor the systems closely to ensure stability.

6. resolved May 21, 2026, 06:30 PM UTC

   All impacted systems are now operating normally, and Two-Way Roleplay functionality has been fully restored. We will continue to monitor the systems closely. The incident is now resolved.

Read the full incident report →

Started May 18, 2026, 01:17 PM UTC· resolved May 18, 2026, 01:55 PM UTC

1. investigating May 18, 2026, 01:17 PM UTC

   The 'Featured Hubs' widget, wherever used on the Mindtickle platform, is not loading. The Hubs and individual media/assets are working properly. We are currently investigating this. All other capabilities of the Mindtickle platform are operating normally.

2. identified May 18, 2026, 01:21 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 18, 2026, 01:31 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 18, 2026, 01:55 PM UTC

   This incident has been resolved. The 'Featured Hubs' widget is now loading as expected, and the Mindtickle platform is fully operational.

Read the full incident report →

Started May 11, 2026, 02:25 PM UTC· resolved May 11, 2026, 04:35 PM UTC

1. investigating May 11, 2026, 02:25 PM UTC

   We are currently investigating an issue impacting call recordings in Call AI. Users may experience failures while recording calls. The team is actively working on identifying the root cause and restoring normal functionality. Further updates will be shared shortly.

2. resolved May 11, 2026, 04:35 PM UTC

   The issue has now been identified and resolved, and Call AI services are operating normally again. We will share a detailed Root Cause Analysis (RCA) shortly.

Read the full incident report →

Started Mar 24, 2026, 03:00 PM UTC· resolved Mar 24, 2026, 03:00 PM UTC

1. resolved Mar 26, 2026, 11:50 AM UTC

   Between 07:50 AM and 11:25 AM PT on March 24, 2026, the Mindtickle platform experienced a service disruption affecting Call Recordings. Full service was restored by 11:25 AM PT. We are currently conducting a thorough investigation and will publish a detailed Root Cause Analysis (RCA) shortly.

2. postmortem Mar 27, 2026, 03:50 AM UTC

   We experienced a service disruption impacting **Call AI bot functionality**. During this time, **select scheduled meetings were affected**, where Call AI bots were unable to join and record. The issue has been fully resolved, and services have returned to normal operation. ‌ **Impact** * Call AI bots were unable to join **select scheduled meetings** * Call recordings and transcriptions were not generated for impacted meetings ‌ **Timeline** * **07:50 AM PT:** Incident began; bots unable to join select meetings * **07:52 AM PT:** Alerts triggered; issue detected and investigation initiated * **09:00 AM PT:** Root cause identified within Call AI bot infrastructure * **10:15 AM PT:** Mitigation measures applied; service recovery in progress * **11:25 AM PT:** Service fully restored; bots resumed normal operation ‌ **Root Cause** This incident was caused by a **transient issue within the Call AI bot infrastructure**, which impacted service availability during the affected period. ‌ **Next Steps** * Exploring & implement enhanced failover mechanisms to improve system resilience * Introducing more granular monitoring and alerting for faster detection ‌ We apologize for the inconvenience caused and appreciate your patience.

Read the full incident report →

Started Feb 19, 2026, 02:33 PM UTC· resolved Feb 19, 2026, 02:52 PM UTC

1. investigating Feb 19, 2026, 02:33 PM UTC

   we are currently investigating an issue affecting the Analytics Platform. While out of the box dashboards remain functional, users may be unable to open existing custom reports or create new ones. Our engineering team is identifying the root cause, and we will provide an update as soon as more information is available. We apologize for the interruption to your workflow.

2. investigating Feb 19, 2026, 02:50 PM UTC

   We are continuing to investigate this issue.

3. resolved Feb 19, 2026, 02:52 PM UTC

   We have successfully resolved the issue affecting the Analytics Platform. All custom reports should now load correctly, and the ability to create new reports has been fully restored. Our team has verified the fix, and we are monitoring the system to ensure continued stability. Thank you for your patience while we worked to get this back online.

Read the full incident report →

Started Feb 12, 2026, 03:28 AM UTC· resolved Feb 12, 2026, 03:37 AM UTC

1. investigating Feb 12, 2026, 03:29 AM UTC

   We are observing high error rates for both admins and end users for select customer instances. We are investigating the issue and will share an update shortly.

2. identified Feb 12, 2026, 03:29 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 12, 2026, 03:36 AM UTC

   We have implemented a fix and are monitoring the progress. We are seeing the error rates coming down.

4. resolved Feb 12, 2026, 03:37 AM UTC

   The incident has been resolved. All admins and end users should be able to access the platform without any concerns.

Read the full incident report →

Started Feb 06, 2026, 07:15 AM UTC· resolved Feb 06, 2026, 07:15 AM UTC

1. resolved Feb 06, 2026, 01:23 PM UTC

   [Resolved] A subset of customers (admins only) experienced intermittent failures when admins attempted to upload content to the platform. End users were not impacted and were able to view existing content without issues. The issue has been resolved, and all admin content upload workflows are operating as expected. Incident Timeline Start Time: 05 Feb 2026, 11:15 PM PT Resolved Time: 06 Feb 2026, 05:12 AM PT We will share a detailed RCA once the post-incident review is complete.

Read the full incident report →

Started Nov 18, 2025, 02:45 PM UTC· resolved Nov 18, 2025, 05:17 PM UTC

1. identified Nov 18, 2025, 02:45 PM UTC

   The Two-Way Roleplay Missions feature is currently experiencing an outage and is inaccessible. This is a direct consequence of the ongoing, widespread Cloudflare network outage. We are closely monitoring Cloudflare's recovery efforts and expect the service to be restored as soon as their network disruption is resolved.

2. monitoring Nov 18, 2025, 02:54 PM UTC

   Cloudflare has confirmed the deployment of a fix for the underlying network issue. We are now observing a significant decrease in error rates on our platform, with Two-Way Roleplay Missions starting to become accessible again. We will continue to monitor the platform closely as services stabilize, and will post a final resolution notice once all operations are confirmed to be fully back to normal.

3. monitoring Nov 18, 2025, 03:37 PM UTC

   Two-Way Roleplay Missions are operational. We will continue to monitor the platform closely to ensure stability before marking this incident as fully resolved.

4. resolved Nov 18, 2025, 05:17 PM UTC

   All services are normal. A detailed Root Cause Analysis (RCA) will be published shortly.

Read the full incident report →

Started Oct 20, 2025, 04:30 PM UTC· resolved Oct 20, 2025, 08:55 PM UTC

1. identified Oct 20, 2025, 04:30 PM UTC

   There has been a second surge of error rates and latencies reported due to an AWS outage in the US-East-1 region. The AWS team is actively working on this and we are monitoring the status closely. Though the platform is up and some features are working, select customers and users may face intermittent failues in accessing the platform. We will keep updating the status page as soon as we receive updates from AWS. We apologize for the inconvenience caused here. AWS Outage Status: https://health.aws.amazon.com/health/status

2. identified Oct 20, 2025, 05:51 PM UTC

   As per AWS, their internal subsystems are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. They are now applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside. We will share an update once the AWS team has applied the mitigations and we have verified at our end.

3. monitoring Oct 20, 2025, 07:43 PM UTC

   Post the recovery and fix from the AWS team, we are observing a significant improvement in the Mindtickle platform performance. Most Mindtickle workflows are now functioning normally. Our teams continue to monitor the situation closely and will provide an update once the issue is fully resolved. AWS Outage Status: https://health.aws.amazon.com/health/status

4. resolved Oct 20, 2025, 08:55 PM UTC

   The incident has been resolved, and the Mindtickle platform is now operating normally. We continue to closely monitor performance to ensure everything remains stable.

Read the full incident report →

Started Oct 20, 2025, 10:07 AM UTC· resolved Oct 20, 2025, 11:03 AM UTC

1. monitoring Oct 20, 2025, 10:07 AM UTC

   Earlier today, some Mindtickle customers experienced high error rates and intermittent service disruptions due to an AWS outage in the US-East-1 region. AWS has since deployed a fix, and we’re observing steady recovery across our systems. Error rates are declining, and most services are returning to normal. Our teams continue to monitor the platform closely to ensure full stability. We’ll share a final update once all systems are fully restored. Thank you for your patience and understanding.

2. resolved Oct 20, 2025, 11:03 AM UTC

   The incident has been resolved, and the Mindtickle platform is operating normally.

Read the full incident report →

Started Oct 16, 2025, 03:54 AM UTC· resolved Oct 16, 2025, 04:50 AM UTC

1. investigating Oct 16, 2025, 03:54 AM UTC

   We’ve identified an issue where specific pages within the Analytics section are displaying stale or incorrect data for select customers. Our team is actively investigating the issue and will share further updates as soon as possible. Impacted Areas: 1. OData Routes (Partial Routes) – Stale data visible 2. Analytics Platform Pages (Series, Modules, Learner & Learner Groups) – Incorrect data displayed 3. Custom Reports shared with customers via Email (in the last 12 hours) – Incorrect data displayed

2. monitoring Oct 16, 2025, 04:29 AM UTC

   A fix has been implemented and we are running a sanity check at our end.

3. resolved Oct 16, 2025, 04:50 AM UTC

   This incident has been resolved. We will share an RCA once the postmortem is complete.

Read the full incident report →

Started Oct 14, 2025, 07:10 AM UTC· resolved Oct 14, 2025, 07:10 AM UTC

1. resolved Oct 15, 2025, 11:19 AM UTC

   Between October 14, 2025, 12:10 AM PT and October 15, 2025, 3:00 AM PT, select Call AI customers experienced a delay in the processing of call recordings. Impact: 1. Call recordings generated during this period were temporarily not visible on the Call Listing page. What Continued to Work: 1. All recordings captured before the incident window remained accessible on the Call Listing page. 2. For calls recorded during the incident, Recording Ready emails were successfully sent to end users, allowing access to the recordings via the email links. We are conducting a post-incident analysis to identify the root cause. A detailed Root Cause Analysis (RCA) report will be shared once the investigation is complete.

Read the full incident report →

Started Sep 24, 2025, 09:01 AM UTC· resolved Sep 24, 2025, 10:09 AM UTC

1. investigating Sep 24, 2025, 09:01 AM UTC

   We're currently experiencing an high error rate in our in-platform analytics and are actively investigating the root cause to restore full functionality.

2. identified Sep 24, 2025, 09:40 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 24, 2025, 09:51 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 24, 2025, 10:09 AM UTC

   A fix has been applied, and the incident has been resolved. We are continuing to monitor this closely.

Read the full incident report →

Started Sep 20, 2025, 06:40 AM UTC· resolved Sep 20, 2025, 07:40 AM UTC

1. resolved Sep 20, 2025, 10:29 AM UTC

   Between 19 Sep 2025, 23:40 to 10 Sep 2025, 00:50 PT, the Mindtickle admin and learning site were unavailable for select users in our US Production region. The platform has recovered and is fully functional now. We will share an RCA post a detailed postportem. Apologies for the inconvenience caused.

2. postmortem Sep 24, 2025, 04:52 AM UTC

   ## Incident Summary On September 19, 2025, during a periodic platform upgrade, the Mindtickle platform experienced an outage lasting approximately 1 hour and 43 minutes. The disruption was caused by a configuration issue on upgraded servers, which led to resource constraints. As a result, application services could not run as expected, causing downtime for a set of customers \(in the US region\). Our engineering team identified the issue, corrected the configuration, and rotated the affected servers. Services were fully restored and stabilized thereafter. * Start time: September 19, 2025, 11:06 PM PT * End time: September 20, 2025, 00:49 AM PT ## Impact * Workflows Impacted: All workflows on the platform * Customers Impacted: A select set of customers \(in the US region\) ## Incident Timeline \(PT\) * September 19, 2025, 11:06 PM: Users began experiencing downtime and errors * September 19, 2025, 11:10 PM: Engineering team detected the issue and initiated an investigation * September 19, 2025, 11:45 PM: Root cause identified \(server resource constraints\); remediation started * September 20, 2025, 00:30 AM: Services began recovering as corrected configurations were applied * September 20, 2025, 00:49 AM: All services confirmed healthy; incident resolved ## Root Cause The outage was caused by misconfigured disk sizes in newly upgraded servers. This resulted in resource shortages that prevented application services from running. This misconfiguration was not detected during pre-upgrade validation because upgrade scripts did not fully account for updated server requirements. ## Preventive Actions To prevent recurrence, we are implementing the following measures: 1. Configuration Management: Standardize and validate server configurations across environments. 2. Upgrade Safeguards: Introduce a staggered approach with cooldown periods between server pool rotations. 3. Runbook Enhancements: Update documentation with environment-specific requirements and lessons learned. 4. Proactive Monitoring: Enhance alerts to detect early signs of resource constraints. We sincerely apologize for the disruption this outage caused. We are committed to learning from this incident and strengthening our upgrade and validation processes to ensure greater reliability and resilience of the Mindtickle platform.

Read the full incident report →

Started Sep 12, 2025, 02:30 PM UTC· resolved Sep 12, 2025, 02:30 PM UTC

1. resolved Sep 12, 2025, 02:42 PM UTC

   Mindtickle was not impacted by the Salesloft Drift data breach, and no customer data was compromised. Mindtickle does not use or integrate with the Drift platform or any other Salesloft products. Our systems, including those that integrate with Salesforce, have not been compromised or pose a similar risk to the Salesloft Drift incident. We are proactively reviewing our third-party vendors and sub-processors. To date, we have not received any notification of incidents or data breaches from any of these third parties. We'll continue to actively monitor for any potential impact and provide updates as appropriate. For questions regarding this update, please contact [email protected].

Read the full incident report →

Started Aug 29, 2025, 07:49 PM UTC· resolved Aug 29, 2025, 07:49 PM UTC

1. resolved Sep 03, 2025, 04:58 AM UTC

   On August 29, 2025, between 12:49 PM and 1:17 PM PT, the Mindtickle platform experienced high error rates. During this period, multiple users may have encountered issues logging into the platform or accessing programs. The incident was resolved at 1:17 PM PT, and services have since been fully restored. We are conducting a detailed analysis of the incident and will share a Root Cause Analysis (RCA) once it is complete. We apologize for any disruption this may have caused and thank you for your patience.

2. postmortem Sep 24, 2025, 04:44 AM UTC

   # **Incident Summary** On August 29, 2025, the Mindtickle platform experienced a temporary disruption where some users were unable to log in or access programs. The issue was identified and resolved within 28 minutes, restoring the platform to normal operation. * Start time: August 29, 2025, 12:49 PM PT * End time: August 29, 2025, 01:17 PM PT # **Impact Area** The following functionality was impacted during the incident: * User logins * Access to programs/assets \(assigned series, modules, and assets were impacted\) # **Incident Timeline** * **August 29, 2025, 12:49 PM PT:** Users began experiencing login and program access errors. * **August 29, 2025, 12:55 PM PT:** The Engineering team detected elevated error rates and initiated an investigation. * **August 29, 2025, 01:17 PM PT:** Corrective actions applied; services restored to a stable state. # **Root Cause Analysis** The disruption was caused by system resource exhaustion in one database cluster, which led to request timeouts and high error rates for affected services. Once identified, the engineering team stabilized services by resetting resource pools and prioritizing critical traffic. # **Next Steps and Preventive Actions** * **System Safeguards:** Implement circuit breakers to isolate and recover from failures faster. * **Resiliency Improvements:** Maintain priority channels for critical operations to reduce customer impact in similar scenarios.

Read the full incident report →

Started Jun 05, 2025, 08:36 AM UTC· resolved Jun 05, 2025, 08:36 AM UTC

1. resolved Jun 05, 2025, 09:12 AM UTC

   On June 5th, 2025, between 1:36 AM PT and 2:06 AM PT, users experienced an issue where the Series and Modules were inaccessible on the Mindtickle platform. The issue was resolved at 2:06 AM PT, and full access to both Series and Modules has been successfully restored. We are actively monitoring the system to ensure continued stability. A detailed Root Cause Analysis (RCA) will be shared shortly.

2. postmortem Jun 13, 2025, 09:30 AM UTC

   **Incident Summary** On June 5, 2025, a temporary issue was observed where users were unable to view the Series Listing Page across the platform. The root cause was traced to a misconfiguration in the infrastructure setup of a new caching service \(Amazon DAX\) introduced to improve system performance. The configuration error blocked communication between application services and the DAX cluster, leading to service disruptions. The issue was promptly identified and resolved by rolling back the change. **Impact Area** The following functionality was impacted during the incident: * Series Listing Page \(site-wide impact\) **Incident Timeline** * June 5, 2025, 1:37 AM PT: Users began experiencing issues with the Series Listing Page. * June 5, 2025, 1:56 AM PT: The engineering team detected the issue and initiated an investigation.June 5, 2025, 2:08 AM PT: Rollback completed and services were restored to a stable state. **Root Cause Analysis** The issue was caused by a misconfiguration in the security settings of the newly introduced Amazon DAX cluster in the production environment. The security group associated with the cluster did not allow required inbound traffic from the application services. This caused service pods to fail and led to errors on the Series Listing Page. **Next Steps and Preventive Actions** * **Improved Health Checks**: Applications will include readiness and liveness probes for external dependencies like DAX to detect failures earlier. * **Enhanced Monitoring and Alerts**: DAX metrics will be integrated into the monitoring framework to ensure real-time visibility and proactive alerts. We apologize for the inconvenience caused by this incident.

Read the full incident report →

Started May 24, 2025, 05:40 AM UTC· resolved May 24, 2025, 06:10 PM UTC

1. resolved Jun 02, 2025, 04:39 PM UTC

   User progress data was not reflected for the specific modules (Quick Update, ILT, Checklist, Quest, and Course) in analytics reports. The impacted reports include; 1. Out-of-the-box (OOTB) reports 2. Self-Serve Reports (SSR) 3. OData feeds 4. Custom dashboards

2. postmortem Jun 02, 2025, 04:39 PM UTC

   ### **Summary** As part of our broader initiative to move away from our legacy system to a more robust platform, we have been migrating components of our analytics setup in phases for the past few months \(starting March 2025\) to ensure stability. On May 24, 2025, we completed the final phase of this migration, transitioning the remaining components to a more modern and reliable infrastructure. Following the migration, we observed a **delay in the reflection of user progress data** for five specific learning module types — **Quick Update, ILT, Checklist, Quest, and Course** — in analytics reports. ### **Impact** User progress data was not reflected for the affected modules in analytics reports. The impacted reports include; * Out-of-the-box \(OOTB\) reports * Self-Serve Reports \(SSR\) * OData feeds * Custom dashboards The issue was limited strictly to the **user’s progress data in analytics**. ### **What Was Not Impacted** * The Rule Engine * The progress status shown on the user dashboard * Real-time tracking during user activity \(e.g., viewing or completing modules\) Learner experience and actual learning progress within the platform were unaffected. ‌ ### **Root Cause** * As part of the migration effort, events from the legacy system were being transitioned into the new platform via a migration pipeline. * However, due to historical lag issues in the legacy system, a subset of user progress events—specifically for five identified module types—had not yet been generated or processed within the legacy system. As a result, these events never reached the migration pipeline. * Despite the pipeline appearing clear for an extended period, the contentious events remained unprocessed within the legacy system and had yet to enter the migration flow, revealing a hidden backlog that challenged initial expectations. ### **Fix Implemented** * After a thorough and rigorous analysis pinpointed the issue to the legacy system, a backfill process was promptly initiated to accurately identify and carefully ingest the missing user progress events into the new system. * This process was completed on **29 May 2025, 11:10 am PT / 11:40 pm IST**. All the user progress data should now be visible correctly on the platform. ### **Safeguards Going Forward** The new system features a reliable and observable data pipeline, built with enhanced fault tolerance and end-to-end monitoring. The pipeline is equipped with automated alerting to ensure data integrity, consistency, and anomaly detection, including scenarios such as missed events, duplicate records, and data lag. We apologize for the inconvenience and remain committed to improving system reliability.

Read the full incident report →

Started May 22, 2025, 06:10 AM UTC· resolved May 22, 2025, 06:44 AM UTC

1. investigating May 22, 2025, 06:10 AM UTC

   Our team is actively investigating the issue and working to restore full access as quickly as possible.

2. identified May 22, 2025, 06:34 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 22, 2025, 06:44 AM UTC

   This incident has been resolved, we will publish a detail root cause analysis soon.

Read the full incident report →