CallTower experienced a critical incident on November 1, 2024 affecting Microsoft Teams - Direct Routing and Microsoft Operator Connect, lasting 2h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 01, 2024, 07:48 PM UTC
CallTower support is investigating multiple reports of issues with inbound calls to Auto Attendants and Call Queues for Microsoft Teams GCC High customers. Engineers are working to identify the root cause of the issue. CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to [email protected]. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.
- identified Nov 01, 2024, 08:21 PM UTC
The issues with inbound calls to Auto Attendants and Call Queues for Microsoft Teams GCC High customers has been identified/confirmed as an issue with Microsoft. CallTower Engineering have a case open with Microsoft that they are actively escalating. CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to [email protected]. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
- identified Nov 01, 2024, 09:01 PM UTC
CallTower Engineering continues to work with Microsoft on the inbound calling issues with Teams GCC High. Here is the update from Microsoft: Title: Callers placing calls to Microsoft Teams hosted auto attendants and call queues may fail User impact: Callers placing calls to Microsoft Teams hosted auto attendants and call queues may be failing. Current status: Our monitoring has identified that a recent service configuration change contained an error, preventing auto attendant and call queues from operating as expected. We're in the process of correcting the configuration to remediate impact. Scope of impact: Any calls placed to auto attendants and call queues hosted on the impacted infrastructure may be impacted. Root cause: A recent service configuration change contained an error, preventing auto attendant and call queues from operating as expected, resulting in impact. Next update by: Friday, November 1, 2024, at 10:00 PM UTC CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to [email protected]. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
- identified Nov 01, 2024, 09:38 PM UTC
CallTower Engineering continues to work with Microsoft on the inbound calling issues with Teams GCC High. CallTower Support has tested with some customers and are getting positive tests back. Microsoft does not have any new updates posted. CallTower has opened a master case for this issue as it is affecting more than one CallTower customer. This issue may or may not directly affect your services. If you are affected, please report the issue to [email protected]. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise.
- resolved Nov 01, 2024, 10:16 PM UTC
CallTower Support has confirmed with the affected customers that the inbound calling issues with Teams GCC High is resolved, and the associated cases will be closed out. Here is the resolution update from Microsoft: Title: Callers placing calls to Microsoft Teams hosted auto attendants and call queues may fail User impact: Callers placing calls to Microsoft Teams hosted auto attendants and call queues may be failing. Final status: We've completed our corrective actions to the affected configuration and validated this has remediated this issue with affected users and service telemetry. Scope of impact: Any calls placed to auto attendants and call queues hosted on the impacted infrastructure may have been impacted. Start time: Friday, November 1, 2024 at 7:58 AM MDT End time: Friday, November 1, 2024 at 3:18 PM MDT Root cause: A recent service configuration change contained an error, preventing auto attendant and call queues from operating as expected, resulting in impact. Next steps: - We're reviewing our configuration update procedures to better identify and prevent similar impact in the future. This is the final update for this event. CallTower will be closing this master case and all associated cases. Within three business days, an Official Root Cause Analysis (RCA) or a Preliminary RCA will be available and provided to customers upon request.