LMS365 experienced a minor incident on December 11, 2023 affecting Learn365 North Europe and Learn365 United States and 1 more component, lasting 11h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 11, 2023, 09:41 AM UTC
During the deployment of the new LMS365 release, we encountered an issue related to updating certificate statuses. They were inaccurately set to 'Expiring' despite being intended to remain 'Valid'. At present, we have pinpointed the root cause of this issue, and our team is actively working on a solution. While this is being addressed, the email notification functionality may experience disruptions, templates for other types of notifications may not be sent (appointment invites, for example). Additionally, we want to notify you that the deployment of the new release in Japan and Australia regions might face potential delays. However, rest assured, we are committed to making every effort to deliver the updates on schedule. We sincerely apologize for any inconvenience this may have caused.
- identified Dec 11, 2023, 03:37 PM UTC
As we continue to address the ongoing issue, we are currently in the process of rolling out the LMS365 December release in the Australia and Japan regions. As previously mentioned, these respective regions have not been affected by the incident other than experiencing delays in the delivery timeline.
- monitoring Dec 11, 2023, 04:13 PM UTC
We are pleased to inform you that the fix for the issue with certificate statuses has been successfully implemented in all regions except for US Government Community Cloud. As a result, the erroneous 'Expiring' statuses have been rectified, and certificates are now correctly displaying as 'Valid'. Additionally, the notification functionality should now be operating properly. We are actively monitoring the results of the fix to ensure stability and accuracy. We appreciate your patience and understanding throughout this process. Should you encounter any issues or have further concerns, please don't hesitate to reach out to our support team. Thank you for your cooperation.
- resolved Dec 11, 2023, 09:20 PM UTC
We would like to inform you that the fix was rolled out for the US Government Community Cloud region as well. We can confirm that the issue is now resolved. We apologize for any inconvenience caused by this incident. If you encounter any issues or have additional concerns, please feel free to contact our support team.
- postmortem Dec 21, 2023, 07:57 AM UTC
We sincerely apologize for the inconvenience caused by the incident surrounding the deployment of the December LMS365 release. Root Cause: During the deployment of the new LMS365 release, code error caused the system to inaccurately update certificate statuses. Resolution: The team identified and rolled out a fix for the issue across affected regions, which have resolved inaccuracies in certificate statuses and restored notification functionality. Remediation items: * Continuous monitoring to ensure system stability and accuracy of the certificate status post-resolution. * We have reviewed deployment processes to prevent similar issues in future releases. We appreciate your understanding and patience as we work diligently to prevent any similar errors.