Teem incident
S2 - Teem calendar error - SSL error "SSL: CERTIFICATE_VERIFY_FAILED]"
Teem experienced a major incident on October 23, 2023 affecting Exchange Sync, lasting 4h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 23, 2023, 01:33 PM UTC
We are currently investigating a calendar issue with Teem. Some customers may see an error on their calendars - "SSL: CERTIFICATE_VERIFY_FAILED]"
- identified Oct 23, 2023, 03:07 PM UTC
The issue with "Teem calendar error - SSL error "SSL: CERTIFICATE_VERIFY_FAILED]" has been identified and a fix is being implemented. We will post another update at 1:07pm MST or sooner.
- monitoring Oct 23, 2023, 03:33 PM UTC
A fix has been implemented. We are moving into the Monitoring Phase for the next 1 hour.
- resolved Oct 23, 2023, 05:56 PM UTC
As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.
- postmortem Oct 27, 2023, 07:50 PM UTC
**Eptura Workplace Detailed Root Cause Analysis | Severity 2 | October 23, 2023** **SSO Certificate Failed** We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. **Description:** On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. **Type of Event:** Performance Degradation **Services\\Modules Impacted:** Production: Calendars **Timeline:** On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. After a short investigation support was able to replicate this issue and all customers were notified of the disruption via Status Page at 9:33am EDT. 11:07am EDT internal teams have identified the issue and began working on the fix. At 11:33am the fix was implemented, and the status page was updated to a monitoring phase. As not additional reports were received, and customers confirmed the resolution the status page was updated to a resolved phase at 1:56pm. **Total Duration of Event:** 2hrs 3mins **Root Cause:** Internal teams found a certificate that expired on the exchange adapter for Teem. **Preventative Action:** Internal teams continue to monitor all systems and have created internal processes to ensure alerts are set up to prevent this from happening in the future.