Teem Outage History

Started May 14, 2025, 04:07 PM UTC· resolved May 16, 2025, 04:58 PM UTC

1. investigating May 14, 2025, 04:07 PM UTC

   We are currently investigating an issue with Teem. We will update you when we have more information.

2. identified May 14, 2025, 06:58 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified May 14, 2025, 07:36 PM UTC

   The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 6 PM CST. If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin From there please reach out to support for help as there is a current workaround.

4. identified May 14, 2025, 10:22 PM UTC

   The issue with the Plugin has been identified and a fix is being implemented. We will post another update tomorrow at 9 AM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

5. identified May 15, 2025, 02:07 PM UTC

   The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 12 PM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

6. identified May 15, 2025, 05:31 PM UTC

   The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 5 PM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

7. identified May 16, 2025, 12:19 AM UTC

   The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 9 AM MST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

8. monitoring May 16, 2025, 02:37 PM UTC

   A fix has been implemented. We are moving into the Monitoring Phase for the next two hours. We appreciate your patience as our team worked diligently to provide a solution for this.

9. resolved May 16, 2025, 04:58 PM UTC

   As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. Thank you for all the patience as we worked through this issue.

Read the full incident report →

Started Apr 14, 2025, 05:15 PM UTC· resolved Apr 21, 2025, 07:34 PM UTC

1. investigating Apr 14, 2025, 05:15 PM UTC

   We are currently investigating an issue with Teem. We will update you when we have more information.

2. identified Apr 15, 2025, 06:26 AM UTC

   Our engineering team has identified inconsistent time zone behavior for all-day bookings via Plugin and is working to validate the impact across all customers. Users can continue to make bookings via https://app.teem.com/ or directly from the device while we work on addressing the issue. We shall keep you informed as soon as more information becomes available. Thank you for your continued patience and understanding.

3. identified Apr 15, 2025, 02:55 PM UTC

   Our engineering team has identified that the issue is occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. Thank you for your continued patience and understanding.

4. identified Apr 15, 2025, 07:07 PM UTC

   Our engineering team continues to work on implementing a solution occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

5. identified Apr 15, 2025, 11:27 PM UTC

   Our engineering team continues to work on implementing a solution occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

6. identified Apr 16, 2025, 03:11 AM UTC

   Our engineering team continues to work on implementing a solution occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

7. identified Apr 16, 2025, 09:46 AM UTC

   We are actively investigating an issue affecting all-day bookings. Our Development team is working on a solution. We'll share more updates as soon as they become available. Thank you for your continued patience and understanding.

8. identified Apr 16, 2025, 07:01 PM UTC

   Our engineering team continues to work on the issue affecting all-day bookings. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

9. identified Apr 16, 2025, 11:00 PM UTC

   Our engineering team has developed a fix, and it is now being tested internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

10. identified Apr 17, 2025, 03:17 AM UTC

    Our QA team are continuing to test this issue internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

11. identified Apr 17, 2025, 07:00 PM UTC

    Our QA team continues to test this issue internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

12. monitoring Apr 21, 2025, 03:09 PM UTC

    A fix has been developed and applied to production. We are actively monitoring the situation to ensure continued stability and performance. If you’re still experiencing any issues, please reach out to our support team through your existing support case on the Customer Hub, or create a new one if none exists. We will continue to monitor the situation for the next 8 hours before marking it as resolved. Thank you for your patience.

13. monitoring Apr 21, 2025, 07:33 PM UTC

    We are continuing to monitor for any further issues.

14. resolved Apr 21, 2025, 07:34 PM UTC

    This issue is currently resolved with a release that happened last night. After confirmation we are going to be closing out this Status Page. We appreciate all the patience with our customer base as we have worked diligently to get this update out.

Read the full incident report →

Started Mar 04, 2025, 02:19 PM UTC· resolved Mar 05, 2025, 06:31 PM UTC

1. investigating Mar 04, 2025, 02:19 PM UTC

   We are currently investigating an issue with TEEM and the inability to access the platform. Our Engineering and Cloud Operations team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 9:48am CST. Thank you for your patience as we work to resolve this issue.

2. investigating Mar 04, 2025, 05:14 PM UTC

   We are currently investigating an issue with TEEM and the inability to access the platform. Our Engineering and Cloud Operations team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 1:00 PM CST. Thank you for your patience as we work to resolve this issue.

3. identified Mar 04, 2025, 05:49 PM UTC

   The issue with performance has been identified and a fix is being implemented. We will post another update at 1PM CST.

4. monitoring Mar 04, 2025, 06:33 PM UTC

   A fix has been implemented. We are moving into the Monitoring Phase for the next 24 hours.

5. identified Mar 04, 2025, 10:35 PM UTC

   The issue with App.teem.com performance has been identified and a fix is being implemented. We will post another update at 10 AM CST.

6. monitoring Mar 05, 2025, 03:49 PM UTC

   A fix has been implemented. We are moving into the Monitoring Phase for the next two hours. If you have any issues please don't hesitate to reach out to support.

7. resolved Mar 05, 2025, 06:31 PM UTC

   As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

8. postmortem Mar 20, 2025, 04:32 PM UTC

   **Teem by Eptura detailed Root Cause Analysis | 3/4/2025** **S1 – Teem Inaccessible** We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** **\(The Incident is logged in MST\)** On March 4th, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm. **Type of Event:** Outage **Services/Modules impacted:** [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring) **Timeline:**` `**The timeline is posted in MST.** Around 7:34 AM. We received notice of some people not able to access [app.teem.com](http://app.teem.com). At 10:19 AM our Engineering team clearing tables to help alleviate performance. At 11:29 AM we moved the issue to an S2 due to website being up and accessible. At 10:17 PM we are finishing clearing tables meaning performance should now be at full. 3:49 AM website is working as intended. At 9:17 AM the issue was resolved and we entered the monitoring phase for two hours. At 11 AM we cleared the status page. **Total Duration of Event:** 27 Hours **Root cause:** * Duplicate event generation: The event processing logic did not prevent the creation of redundant events. * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time. * Push Callback Timing Issue: * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end. * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed. * This resulted in the original event remaining incomplete and missing updates from Outlook. * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created. **Remediation:** To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates. **Preventative Action:** To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

Read the full incident report →

Started Feb 28, 2025, 08:15 PM UTC· resolved Mar 05, 2025, 12:42 AM UTC

1. identified Feb 28, 2025, 08:15 PM UTC

   We have successfully completed maintenance to enhance security and compliance by transitioning to Microsoft Graph API for calendar services using Microsoft Exchange. We are aware of a small issue that occurred during the release we are addressing, which may cause calendars to produce duplicate meetings. Our team is actively managing the operational incident and working to resolve it as quickly as possible. If you are experiencing an issue with duplicate calendar events, you can temporarily remove the Location from the calendar to prevent duplication notices. Once we implement the update early next week, you can re-add the Location. Steps to remove the Location: Go to your calendar section on app.teem.com on the left side of the page. Select the proper account, then navigate to the calendar experiencing the duplication issue. Under the Location option click the red dot, this will remove the location from that calendar, halting the duplicate notifications. After our team implements the update early next week, you can re-add the Location. We will keep you informed once the update has been implemented. For any questions or to learn more about alternative options to Microsoft Graph API, please review the technical documentation on our Knowledge Center or contact support. We will update the status page as our Engineering team works through this issue on Monday March 3rd at 8 AM MST.

2. identified Mar 03, 2025, 04:27 PM UTC

   Important update on calendar syncing We recently completed maintenance to enhance security and compliance by transitioning to Microsoft Graph API for calendar services using Microsoft Exchange. We are aware of a small issue that occurred during the release, which may cause calendars to produce duplicate meetings. Our team is actively managing the operational incident and will release the fix today, March 3, 2025, at 10:30 PM MST. This fix will address known issues for Google Sync, as well as performance and O365 Calendaring issues. If you are experiencing an issue with duplicate calendar events, you can temporarily remove the Location from the calendar to prevent duplication notices. Once we implement the update, you can re-add the Location. Steps to remove the Location: Go to your calendar section on app.teem.com on the left side of the page. Select the proper account, then navigate to the calendar experiencing the duplication issue. Under the Location option click the red dot, this will remove the location from that calendar, halting the duplicate notifications. After our team implements the update early next week, you can re-add the Location. We will keep you informed once the update has been implemented. For any questions or to learn more about alternative options to Microsoft Graph API, please review the technical documentation on our Knowledge Center or contact support. This will be completed at 10:30 PM MST. This will fix known issues for Google Sync as well as performance issues and O365 Calendaring issues.

3. resolved Mar 05, 2025, 12:42 AM UTC

   As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

Read the full incident report →

Started Feb 25, 2025, 03:03 PM UTC· resolved Feb 25, 2025, 09:01 PM UTC

1. investigating Feb 25, 2025, 03:03 PM UTC

   We are currently investigating an issue with the loading of app.teem.com. Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 11 AM MST

2. investigating Feb 25, 2025, 05:29 PM UTC

   We are currently still investigating an issue with app.teem.com and logging Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 12:30 PM MST.

3. investigating Feb 25, 2025, 07:19 PM UTC

   We are currently still investigating an issue with app.teem.com and logging Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 2:30 PM MST.

4. monitoring Feb 25, 2025, 07:51 PM UTC

   A fix has been implemented. We are moving into the Monitoring Phase for the next hour. We will be closing this out at 2 PM MST

5. resolved Feb 25, 2025, 09:01 PM UTC

   As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

6. postmortem Mar 20, 2025, 04:33 PM UTC

   **Teem by Eptura detailed Root Cause Analysis | 2/25/2025** **S1 – Duplicate Events causing downtime** We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** **\(The Incident is logged in MST\)** On February 22, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm. **Type of Event:** Outage **Services/Modules impacted:** [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring) **Timeline:**` `**The timeline is posted in MST.** Around 8:58 AM. We received notice of some instances of slowed experience and inability to access [app.teem.com](http://app.teem.com). At 8:59 AM our Engineering team is reaching out to AWS team to address this. At 10:30 AM we moved the issue to an S1. At 10:45 AM the site became accessible again, but with slowness. At 12:52 PM we resolved this issue and went into a monitoring phase. At 2:02 PM we cleared the status page. **Total Duration of Event:** 4 Hours **Root cause:** * Duplicate event generation: The event processing logic did not prevent the creation of redundant events. * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time. * Push Callback Timing Issue: * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end. * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed. * This resulted in the original event remaining incomplete and missing updates from Outlook. * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created. **Remediation:** To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates. **Preventative Action:** To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

Read the full incident report →