Teem experienced a major incident on September 8, 2020 affecting Exchange Sync, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 08, 2020, 01:39 PM UTC
We are aware of an issue affecting some customers using Office 365 calendar sync for room calendars. An authentication error presents itself when attempting to sync room calendars or authenticate using an Office 365 account on the Teem platform.
- investigating Sep 08, 2020, 06:39 PM UTC
Our engineering team is continuing to work diligently in identifying the root cause of these issues so that we can provide a solution immediately. We will provide updates as they become available.
- investigating Sep 08, 2020, 09:13 PM UTC
Teem has determined that connections to the EWS API for Office365 clients are unavailable and the team is working with Microsoft to determine the root cause and resolution of the issue.
- resolved Sep 09, 2020, 01:39 PM UTC
The issue was fixed and normal operations resumed at approximately 7:00pm MT on Sept. 8. Teem has been monitoring sync traffic since that time. After working with vendors to eliminate possible problems and bottlenecks, a subtle networking issue with the hosting platform itself was discovered. The issue left the subsystems that handle calendar synchronization for Office 365 and Exchange clients in a partially working state with most networking successfully transacting but many or our O365 clients heavily affected. Replacement VMs were created, tested and deployed to clear any underlying issues with the networking, hypervisor or hardware.