Teem incident

S2 - We are investigating intermittent syncing issues with Outlook / Google Calendars

Minor Resolved View vendor source →

Teem experienced a minor incident on August 29, 2023 affecting Google Apps Calendar and Exchange Sync, lasting 6h 15m. The incident has been resolved; the full update timeline is below.

Started
Aug 29, 2023, 08:45 AM UTC
Resolved
Aug 29, 2023, 03:01 PM UTC
Duration
6h 15m
Detected by Pingoru
Aug 29, 2023, 08:45 AM UTC

Affected components

Google Apps CalendarExchange Sync

Update timeline

  1. investigating Aug 29, 2023, 08:45 AM UTC

    We are currently investigating an issue with Teem. We will update you when we have more information.

  2. investigating Aug 29, 2023, 11:55 AM UTC

    We are continuing to investigate this issue.

  3. identified Aug 29, 2023, 02:26 PM UTC

    The issue has been identified and a fix is being implemented.

  4. monitoring Aug 29, 2023, 02:36 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Aug 29, 2023, 03:01 PM UTC

    This incident has been resolved.

  6. postmortem Sep 12, 2023, 06:05 PM UTC

    **Teem Detailed Root Cause Analysis – Severity 2 – 08/29/2023** **Description:** On August 29, 2023, at approximately 4:45am EDT, customer support started to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow. **Type of Event:** Service Disruption **Services\\Modules Impacted:** Production – Calendars **Timeline:** On August 29, 2023, at approximately 4:45am EDT, customer support began to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow. At approximately 10:26am EDT, the engineering team identifies the issue and begins to implement the fix. At 10:36am EDT all customers are made aware that the fix has deployed to production and that we are moving into a monitoring phase. Customers begin to confirm the fix and at 11:01 EDT, the status page is marked as resolved. ‌ **Total Duration of Event:** **6hrs 16mins** ‌ **Root Cause:** The team continues to narrow down the root cause. However, the DevOps team identifies an internal configuration issue and begins to apply a manual fix. **Preventative Action:** Our internal teams are working on enhancing the observability to bring awareness to the appropriate teams.