Search related activities experiencing intermittent errors and slowness

Update timeline

1. identified Mar 20, 2023, 08:22 PM UTC

   Users may be experiencing degraded performance for the following actions: - Searching for contacts or conversations: may be giving errors or returning results in a delayed manner or returning stale results - Creating conversations: maybe return errors or experiencing a lag when searching for contacts - Sending bulk messages: may be experiencing a lag where stuck in Processing state for a while or just taking longer to complete - Opening contact lists: may be returning errors or experiencing a lag in returning results Phone calls, video calls, exchanging messages (fax, secure, email or SMS) is NOT impacted and operating normally. We have identified the reason for the degraded performance and are actively working on resolving the issue.

2. identified Mar 20, 2023, 08:22 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Mar 20, 2023, 09:11 PM UTC

   System is mostly recovered and we will continue to monitor to ensure that we do not see degraded performance for any actions. There are still some stale search results being returned for contact and conversation search, but other than that, the system is fully operational at the moment.

4. resolved Mar 20, 2023, 11:08 PM UTC

   System is fully operational. Summary: From 12:52 PT to 13:57 PT on March 20 2023, the following actions experienced degraded performance: - Searching for contacts or conversations: may be giving errors or returning results in a delayed manner or returning stale results - Creating conversations: maybe return errors or experiencing a lag when searching for contacts - Sending bulk messages: may be experiencing a lag where stuck in Processing state for a while or just taking longer to complete - Opening contact lists: may be returning errors or experiencing a lag in returning results