S2 - O365 calendar Errors - 401/The parameter count cannot be less than zero errors

Update timeline

1. investigating Mar 08, 2023, 04:39 PM UTC

   We are currently investigating an issue with O365 Calendar Errors - Customers are receiving emails about their calendar sync being disabled. While on the Teem platform errors also appear on the resource calendar such as "The parameter count cannot be less than zero errors" or "Too many authorization attempts" and "401 Unauthorized".

2. identified Mar 08, 2023, 05:16 PM UTC

   Our engineering team has identified the issue and will be implementing a fix shortly. We appreciate your patience through this disruption.

3. monitoring Mar 08, 2023, 05:58 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Mar 08, 2023, 08:20 PM UTC

   In efforts to resolve this issue, our team found the root cause to be related to a hotfix that was released on the evening of Monday, March 6, 2023. The hot fix was released to resolve an auto refresh issue that involved calendar credentials, thus causing 401 unauthorized errors. An internal decision was made to roll back the hot fix as there were continual retries and caused source systems (O365) to lock accounts. Some accounts are being unlocked based on source system timing and others may need to be unlocked manually. We are continuing to monitor this issue. Please let our support team know if you continue to experience these errors after you have unlocked your source system accounts.

5. resolved Mar 09, 2023, 12:33 AM UTC

   After monitoring, our support team has received confirmation from multiple customers, after verifying their Source Systems are no longer locked, forcing a re-sync on the calendars are working as designed with out error. If you have additional questions, please reach out to our support team @ [email protected]. A detailed RCA will be followed within 10 business days. We appreciate your patience during this service disruption.