Some users are experiencing longer than expected connection times

Update timeline

1. resolved Jun 13, 2023, 04:18 PM UTC

   Type: Incident Duration: 21 hours and 43 minutes Affected Components: Embedded API, Meeting rooms Jun 13, 18:10:29 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Based on early results connection times and recording/streaming events should be working as expected. UPDATE: We will continue monitoring until Wednesday 14 June 2023, but all systems should be operational and working as expected. Jun 13, 16:18:17 GMT+0 - Investigating - We are currently investigating longer connection times and failed join events to rooms. Cloud recording and live-streaming may also fail to start or take longer to start. Jun 14, 14:01:23 GMT+0 - Resolved - After monitoring overnight, we are resolving this incident. ## Cause The run time for one of our scheduled job’s became longer than it was scheduled for, which caused a spike in one of our databases load. This led to slow performance of that specific database and caused the performance degradation. ## Prevention Steps - We have improved that scheduled job’s performance - We have implemented new alarms to be early notified in case of future similar occurrences and be able to solve the issue without any customer impact. Jun 15, 20:15:21 GMT+0 - Resolved - The team has compiled a [complete post-mortem](https://docs.google.com/document/d/1Rjl2NnMdW7aPhnhEkBhE-5RID89PxKHcB-zNhxba-_U/edit?usp=sharing) of the incident, which is available for viewing at the linked document.