LMS365 Outage History

Started Aug 07, 2025, 01:30 AM UTC· resolved Aug 07, 2025, 01:46 AM UTC

1. resolved Aug 13, 2025, 08:41 AM UTC

   Incident Report – Service Bus Misconfiguration (GCC Region) Date Range: 07 August 2025, 03:46 UTC – 08 August 2025, 09:35 UTC (06 August, 11:46 PM ET - 08 August 2025, 05:35 AM ET) Affected Services: Audit Service, Import-Export Services, Learn365 Notifications, Learn365 Admin Center, Webhooks Subscriptions Region: GCC __________________________________________________ Description A misconfiguration in Azure Service Bus services in the GCC region caused a disruption to several Learn365 components. During the affected period, some services could not process or deliver messages as expected. This impacted the flow of audit information, delivery of notifications, certain administrative actions, and webhook event delivery. ___________________________________________________ Impact - Audit data could not be collected during the outage window. - Import and export operations, including importing training from providers such as Go1 and LinkedIn, and copying courses between catalogs, were delayed or failed. - Email notifications for some events were not sent. - Administrative actions in course settings and user management were partially disrupted: - Deleting a group did not automatically remove its role assignments. - Adding members to a group did not immediately give them the roles associated with that group (some received them later). - Removing members from a group did not remove their associated roles. - Creating, deleting, or permanently removing a user did not trigger webhook notifications for subscribers. - When deleting a user, any groups they belonged to did not have their roles unassigned as expected. - No completion records or created course content were lost. ______________________________________________________ Root Cause A configuration issue in Service Bus topic/subscription management caused message delivery failures in the affected services. ______________________________________________________ Resolution The configuration was corrected, and normal service was restored at 09:35 UTC on 08 August 2025. ______________________________________________________ Next Steps We are strengthening our monitoring and alerting systems to detect and prevent similar issues in the future. This includes: - Improved quota tracking. - Automated subscription health checks. - Early-warning thresholds for message flow disruptions.

Read the full incident report →

Started Jul 21, 2025, 06:51 PM UTC· resolved Jul 21, 2025, 07:37 PM UTC

1. investigating Jul 21, 2025, 06:51 PM UTC

   We are currently investigating an issue affecting GCC and GCCH customers related to user progress not loading from the Learn365 Catalog Admin Center.

2. investigating Jul 21, 2025, 06:51 PM UTC

   We are continuing to investigate this issue.

3. resolved Jul 21, 2025, 07:37 PM UTC

   Our team has resolved the issue preventing User Progress from loading in the Catalog Admin Center. If you continue to experience issues, please reach out to the Zensai Support Team.

Read the full incident report →

Started Jun 30, 2025, 11:17 AM UTC· resolved Jun 30, 2025, 01:13 PM UTC

1. investigating Jun 30, 2025, 11:17 AM UTC

   We are currently investigating an issue affecting Learn365 installations. Our engineering team is actively working to identify the root cause and implement a solution as quickly as possible. We understand the importance of this service and apologize for any inconvenience this may cause. Further updates will be provided here as soon as more information becomes available. Thank you for your patience and understanding.

2. resolved Jun 30, 2025, 01:13 PM UTC

   We have investigated the issue and determined that no changes were required on our end, as it appears to have been a transient incident. The system is currently operating normally. We will continue to monitor the situation to ensure stability. Thank you for your patience.

Read the full incident report →

Started May 30, 2025, 03:57 PM UTC· resolved May 30, 2025, 04:14 PM UTC

1. resolved May 30, 2025, 04:08 PM UTC

   On May 30, 2025, we experienced performance issues with the database servicing pro.weekly10.net. The database was overloaded, leading to long load times and errors for users. To resolve this, we increased the processing power behind the database, resulting in a brief downtime from 3:57 PM GMT to 4:14 PM GMT. The upgrade was completed successfully, and performance improved. We apologize for any inconvenience caused.

Read the full incident report →

Started May 28, 2025, 08:31 AM UTC· resolved May 29, 2025, 07:40 AM UTC

1. identified May 28, 2025, 08:31 AM UTC

   We are currently experiencing an issue affecting certificates added to courses configured in non-English locales. While certificates appear correctly in the progress section — showing validity and allowing download — the files open as blank documents. Users may also see a prompt asking whether to save changes when closing the file. Our team has successfully reproduced the issue and is actively working on a fix. We will share updates here as soon as more information becomes available. Thank you for your patience and understanding.

2. resolved May 29, 2025, 09:37 AM UTC

   We’re pleased to inform you that the issue affecting certificates in non-English locale courses has been successfully resolved. This issue has now been fixed, and the solution has been deployed to all affected environments. We appreciate your patience while our team worked on this, and we’re here if you have any further questions or concerns: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support

Read the full incident report →

Started May 27, 2025, 08:04 AM UTC· resolved Jun 12, 2025, 12:30 PM UTC

1. investigating May 27, 2025, 08:04 AM UTC

   We are currently experiencing an issue where course information from Udemy is not being received by the Learn365 integration. Our team is actively working with Udemy to investigate and resolve the problem. We understand the importance of this integration and are treating this with high priority. Updates will be provided here as the investigation progresses. Thank you for your patience and understanding.

2. resolved Jun 12, 2025, 12:30 PM UTC

   We are glad to inform you that, in collaboraton with the Udemy Team, the issue has been solved and the integration between Learn365 and Udemy is working as intended. Thank you for your patience!

Read the full incident report →

Started May 22, 2025, 09:44 AM UTC· resolved May 23, 2025, 09:34 AM UTC

1. investigating May 22, 2025, 09:44 AM UTC

   We are currently investigating an issue affecting the ability to download certificates and our team is actively working to identify the root cause. Our engineering team is actively analyzing the situation and working to resolve it as quickly as possible. We will provide updates here as soon as more information becomes available. Thank you for your patience and understanding.

2. identified May 22, 2025, 02:52 PM UTC

   We would like to inform you that an issue has been identified. The updates for the Japan and Australia regions have already been successfully deployed. Deployment for the remaining regions is scheduled for tomorrow morning. Please rest assured that we are doing our utmost to minimize any inconvenience. We sincerely appreciate your patience, understanding, and ongoing support throughout this process.

3. resolved May 23, 2025, 09:34 AM UTC

   Root Cause: The download method used a GET request that passed an authentication token and user-specific data as part of the query string. The default server limit for query string length was approximately 2,000 characters. Since the length varied based on user information, some requests exceeded this limit, causing them to fail. Resolution: The server’s query string character limit was increased to accommodate longer requests. This change restored functionality for all affected users. We apologize for the inconvenience this caused and appreciate your patience as we worked to resolve the issue. We remain committed to improving the reliability and resilience of our services.

Read the full incident report →

Started Apr 03, 2025, 03:50 PM UTC· resolved Apr 15, 2025, 08:00 AM UTC

1. identified Apr 03, 2025, 03:50 PM UTC

   We are aware of an issue regarding PDF files failing to display in a Course. We can confirm this is related to a known bug in SharePoint, and Microsoft is actively working to resolve the issue. You can follow the discussion via the URL below. We will continue to update our status page as relevant information becomes available. https://github.com/SharePoint/sp-dev-docs/issues/10180

2. resolved Apr 15, 2025, 01:14 PM UTC

   We would like to let you know that we have implemented our own solution for this problem. The PDF viewer has now been updated to an embeddable version. Please note that with this new player, there is no native support for downloading or printing files. To ensure everything works smoothly, we recommend clearing your cache and cookies before testing the updated viewer. If you run into any further issues or have any questions, please don't hesitate to reach out to our support: team.https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support Thank you for your understanding and patience!

Read the full incident report →

Started Mar 03, 2025, 04:03 PM UTC· resolved Mar 04, 2025, 06:00 AM UTC

1. identified Mar 03, 2025, 04:03 PM UTC

   We can initiate an incident report indicating that customers who do not have the Skills Framework included in their license may experience difficulties loading web parts. Our team is currently addressing this issue and is working on a fix that will be deployed as soon as possible.

2. monitoring Mar 03, 2025, 06:03 PM UTC

   We are pleased to announce that the web part loading issue has been resolved. We appreciate your understanding and patience during this time. If you encounter any additional issues or have any concerns moving forward, please do not hesitate to reach out to us: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support. Your feedback is important to us.

3. monitoring Mar 03, 2025, 06:20 PM UTC

   As a temporary solution, we have modified the configuration of the API Gateway to alleviate issues surrounding loading of webparts. Our team is continuing to work on a permanent solution, with an expected rollout of the next 24 hours

4. resolved Mar 04, 2025, 09:35 AM UTC

   We sincerely appreciate your patience as we addressed the issue. After thorough monitoring and analysis, we are pleased to confirm that the solution we implemented has been successful. If you continue to experience any difficulties, we recommend clearing your browser’s cache and cookies, as this can often resolve lingering problems. If the issue persists after that, please do not hesitate to reach out for further assistance: https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support Thank you once again for your understanding!

5. postmortem Mar 11, 2025, 09:13 AM UTC

   **Incident Postmortem: Learn365 Web Part Loading Issue** ‌ **Impact** Customers who did not have the Skills Framework included in their license experienced issues with loading Learn365 web parts. The issue led to degraded functionality, preventing affected users from fully utilizing the platform. **Root Cause** A code error caused unnecessary requests to the Skills service, even for customers who did not have the Skills Framework feature enabled. This led to failures in web part loading for those users. **Resolution** The issue was resolved by implementing a code fix to eliminate unnecessary requests to the Skills service for customers who do not have this feature in their license. After deployment, monitoring confirmed that web parts were loading correctly for all users. However, in some cases, clearing the browser cache was required for the fix to take effect, which was not the most ideal situation. **Remediation Steps** 1. **Code Fix:** Updated logic to prevent unnecessary calls to the Skills service for non-Skills Framework users. 2. **Quality Assurance Improvements:** Expanded test coverage to include different subscription configurations to catch similar issues before deployment. We sincerely appreciate our customers' patience and understanding during this time and apologize for the inconvenience.

Read the full incident report →

Started Jan 08, 2025, 09:00 AM UTC· resolved Jan 08, 2025, 09:00 AM UTC

1. resolved Jan 08, 2025, 09:00 AM UTC

   We can confirm that yesterday (Tuesday, January 7) there was an issue with the Compliance Service between 8 AM and 11 AM UTC. Right now it is resolved. If you notice any issues going forward, please reach out to us.

Read the full incident report →

Started Dec 09, 2024, 03:55 PM UTC· resolved Dec 10, 2024, 04:20 PM UTC

1. investigating Dec 09, 2024, 03:55 PM UTC

   We are aware of a potential issue that could affect customers in the UK region. We are actively analyzing the situation and will provide a status update as our investigation continues.

2. monitoring Dec 10, 2024, 09:42 AM UTC

   We have implemented a fix for the issue and are actively monitoring the situation. We sincerely apologize for any inconvenience caused and remain committed to delivering reliable services.

3. resolved Dec 10, 2024, 04:20 PM UTC

   This incident has been resolved.

4. postmortem Dec 11, 2024, 01:37 PM UTC

   **Incident Summary** On December 9th, we identified an issue where a bulk enrollment operation of big AD groups triggered the User Service and caused performance degradation for other services hosted on the same App Service Plan. The root cause was an error in our code that allowed the User Service to operate inefficiently under certain conditions. We identified the issue swiftly and rolled out a fix the same day to all regions to prevent further occurrences. **Impact** * **Region Affected**: UK * **Performance Degradation Periods**: * The most significant impact occurred on Monday between **10:59 and 11:26 UTC**, during which the system was still accessible but experienced higher-than-normal loading times. **Resolution** * **Immediate Fix**: * Updated the User Service code to handle enrollment operations more efficiently. * Deployed the fix to all regions to mitigate potential impact elsewhere. * **Monitoring**: * Our existing monitoring systems quickly pinpointed the source of the problem, enabling us to act swiftly. **Mitigation Steps** To reduce the likelihood of similar incidents in the future, we are taking the following steps: **Improved QA Protocols:** Enhance testing for high-load scenarios, particularly for bulk operations. **Monitoring System Enhancements:** Implement additional resource usage alerts to detect and isolate high-impact operations earlier. We sincerely apologize for the inconvenience caused to our users, particularly in the UK region. We deeply regret any disruption this may have caused and are committed to learning from this incident to serve you better in the future.

Read the full incident report →

Started Dec 06, 2024, 12:11 PM UTC· resolved Dec 06, 2024, 04:58 PM UTC

1. investigating Dec 06, 2024, 04:00 PM UTC

   We have identified an issue that is affecting the loading of the web parts in SharePoint. At the moment the issue is no longer affecting our customers. We will add more details as the investigation proceeds.

2. resolved Dec 06, 2024, 04:58 PM UTC

   This incident has been fully resolved. A full postmortem will follow.

3. postmortem Dec 09, 2024, 12:16 PM UTC

   ### Incident Overview During a one-time renewal deployment process for front-end assets, a global configuration update of Azure Front Door was required across both the **general Azure platform** and the **US GCC platform**. While the changes were successfully applied to the general Azure platform, the **US GCC platform encountered issues** due to platform-specific configuration differences. **It is important to emphasize that service degradation was not expected** during this process. However, incomplete propagation of configuration changes on the US GCC platform necessitated additional manual adjustments, resulting in temporary disruption loading the Course Catalog, Course Homepage and My Training Dashboard web parts ### Resolution Manual adjustments were made to complete the deployment on the US GCC platform, and the issue has been resolved. Full services have been restored, and the affected platform is operating normally. ### Follow-Up Actions 1. **Process Improvement:** Deployment processes have been revised to account for platform-specific differences, ensuring smoother execution in the future. 2. **Documentation Update:** Comprehensive updates have been made to include detailed guidance for handling deployments across multiple Azure platforms. 3. **Mitigation for Future Incidents:** While this was a one-time process that will not recur, the lessons learned will be applied to similar scenarios to ensure no unexpected service degradation occurs in the future. We sincerely apologize for any inconvenience caused and remain committed to delivering reliable services.

Read the full incident report →

Started Nov 12, 2024, 04:44 PM UTC· resolved Nov 12, 2024, 06:21 PM UTC

1. investigating Nov 12, 2024, 04:44 PM UTC

   We are investigating an issue impacting some Government Community Cloud High customers where an error message displays when loading web parts on Course Catalogs.

2. monitoring Nov 12, 2024, 05:07 PM UTC

   A fix has been implemented and we will continue to monitor performance.

3. resolved Nov 12, 2024, 06:21 PM UTC

   This incident has been fully resolved.

Read the full incident report →

Started Oct 31, 2024, 05:55 PM UTC· resolved Oct 31, 2024, 06:54 PM UTC

1. investigating Oct 31, 2024, 05:55 PM UTC

   We are currently investigating an issue in the US region where some users can not access the Learn365 Admin Center.

2. monitoring Oct 31, 2024, 06:20 PM UTC

   We have identified the issue and are monitoring it to ensure this behavior is corrected.

3. resolved Oct 31, 2024, 06:54 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 28, 2024, 04:17 PM UTC· resolved Oct 28, 2024, 05:32 PM UTC

1. monitoring Oct 28, 2024, 04:17 PM UTC

   We are currently monitoring a fix for issues related to course creation, course enrollments, and catalog creation in Learn365 for GCCL and GCCH customers.

2. resolved Oct 28, 2024, 05:32 PM UTC

   This incident has been fully resolved. If you have any further issues, please reach out to the Zensai support team. https://helpcenter.zensai.com/hc/en-us/articles/14197648878237-How-to-contact-Zensai-Product-Support

Read the full incident report →