Incident Report – Service Bus Misconfiguration (GCC Region)

Update timeline

1. resolved Aug 13, 2025, 08:41 AM UTC

   Incident Report – Service Bus Misconfiguration (GCC Region) Date Range: 07 August 2025, 03:46 UTC – 08 August 2025, 09:35 UTC (06 August, 11:46 PM ET - 08 August 2025, 05:35 AM ET) Affected Services: Audit Service, Import-Export Services, Learn365 Notifications, Learn365 Admin Center, Webhooks Subscriptions Region: GCC __________________________________________________ Description A misconfiguration in Azure Service Bus services in the GCC region caused a disruption to several Learn365 components. During the affected period, some services could not process or deliver messages as expected. This impacted the flow of audit information, delivery of notifications, certain administrative actions, and webhook event delivery. ___________________________________________________ Impact - Audit data could not be collected during the outage window. - Import and export operations, including importing training from providers such as Go1 and LinkedIn, and copying courses between catalogs, were delayed or failed. - Email notifications for some events were not sent. - Administrative actions in course settings and user management were partially disrupted: - Deleting a group did not automatically remove its role assignments. - Adding members to a group did not immediately give them the roles associated with that group (some received them later). - Removing members from a group did not remove their associated roles. - Creating, deleting, or permanently removing a user did not trigger webhook notifications for subscribers. - When deleting a user, any groups they belonged to did not have their roles unassigned as expected. - No completion records or created course content were lost. ______________________________________________________ Root Cause A configuration issue in Service Bus topic/subscription management caused message delivery failures in the affected services. ______________________________________________________ Resolution The configuration was corrected, and normal service was restored at 09:35 UTC on 08 August 2025. ______________________________________________________ Next Steps We are strengthening our monitoring and alerting systems to detect and prevent similar issues in the future. This includes: - Improved quota tracking. - Automated subscription health checks. - Early-warning thresholds for message flow disruptions.