Teem incident
S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar
Teem experienced a minor incident on June 7, 2023 affecting Google Apps Calendar and Exchange Sync, lasting 3h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 05, 2023, 05:43 PM UTC
We are currently investigating this issue.
- investigating Jun 05, 2023, 08:17 PM UTC
Our team continues to investigate "S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar". We appreciate your patience as we analyze this.
- investigating Jun 05, 2023, 11:22 PM UTC
The team continues to investigate this issue. We appreciate your patience as we work through this.
- investigating Jun 06, 2023, 11:37 AM UTC
We are continuing to investigate this issue.
- investigating Jun 06, 2023, 01:41 PM UTC
The team continues to work through this complex issue involving S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar. We are still in an investigating phase and will continue to provide updates throughout the day.
- identified Jun 06, 2023, 03:10 PM UTC
The issue with S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar has been identified. The team is working on a fix and we will continue to keep you posted on the progress. Thank you and we appreciate your continued patience.
- identified Jun 06, 2023, 05:19 PM UTC
We continue to work toward a resolution for the issue, S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar.
- identified Jun 06, 2023, 07:58 PM UTC
We continue to work toward a resolution for the issue, S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar.
- monitoring Jun 06, 2023, 09:30 PM UTC
A fix has been implemented. We are moving into the Monitoring Phase. If you reported a support ticket, our team will be reaching out shortly to confirm the resolution. Thank you for working with us during this disruption.
- monitoring Jun 07, 2023, 01:41 PM UTC
Since the fix was implemented your external calendar events have been synced, we continue to monitor the calendar events update queue internally. You may experience a delay receiving new calendar events as the queue continues to process.
- monitoring Jun 07, 2023, 04:47 PM UTC
Calendar events that were created before the fix was implemented have been successfully processed. As new calendar events are being created, our engineering and dev ops team continue to monitor the calendar events updated queue.
- monitoring Jun 07, 2023, 07:24 PM UTC
As we continue to monitor the calendar events queue, we are seeing newer events processing in a timely manner. If you are seeing a small delay in new events syncing to source calendar, using the "Calendar Refresh" functionality in Teem will assist for quicker processing of newer calendar events.
- monitoring Jun 07, 2023, 11:15 PM UTC
As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.
- resolved Jun 15, 2023, 04:52 PM UTC
This incident was resolved, Wednesday June 7, 2023 (17:15 MDT). The RCA will follow shortly.
- postmortem Jun 15, 2023, 04:52 PM UTC
**Teem Detailed Root Cause Analysis – Severity 2 – June 5, 2023** **Outlook and Google Events are not Syncing from Teem to Source Calendar** **Description:** _On June 5, 2023, at approximately 11:43am MST, customers support received additional reports of “Outlook and Google Calendar Events are not Syncing from Teem to Source Calendar”. An event created in Teem should sync back to the source calendar and vice versa._ **Type of Event:** _Performance Degradation_ **Services\\Modules Impacted:** _Production \(Reservation Module\)_ **Timeline:**` ` _On June 5, 2023, at approximately 11:43am MST, customer support received additional reports of “Outlook and Google Calendar events are not Syncing from Teem to Source Calendar”. The initial ticket was escalated to a S2 incident at 11:43am MST and posted to our status page alerting customers. Internal teams continue to investigate this issue through June 6, 2023. Our internal team identified the root cause of the issue at 9:10am MST and began work on the resolution. The status page was updated from investing to an identified phase. At 3:30pm MST a hotfix was implemented, and the status page was moved to a monitoring phase. At 4:27pm MST after the hotfix was implemented, internal teams acknowledged that calendar events are slowly being processed through the queue. The internal teams continue to monitor the queue through June 7, 2023. At 6:56am MST, all prior events created before the hotfix was implemented have been cleared. Internal teams continue to monitor new events that were created after the hotfix was released. As monitoring continued, customers reported that all calendar events are syncing at about 4:02pm MST. At 5:15pm MST, the incident was marked as resolved._ **Total Duration of Event:** _2 days 5 hours 32 mins_ **Root Cause Analysis:** _Engineering teams found this was related to a release pushed out May 19, 2023._ **Preventative Action:** _While the engineering team fixed the regression bug. They found that there was an internal process during the release roll out that needed improvement with QA._