Slow Playbooks performance and issues with both inbound and outbound calls completing.

Update timeline

1. investigating Dec 01, 2021, 05:08 PM UTC

   We are aware of ongoing issues that are affected both inbound and outbound calls made through Playbooks. We are investigating the specific cause and will continue to update you through this medium.

2. identified Dec 01, 2021, 05:53 PM UTC

   We continue to investigate the issue and have identified the particular problem.

3. identified Dec 01, 2021, 07:52 PM UTC

   We have identified and implemented a temporary fix for this issue. We will be deploying a permanent fix between 0300-0400 UTC on December 2nd 2021 and will post further updates through this forum after this deployment.

4. monitoring Dec 02, 2021, 06:00 AM UTC

   A fix was deployed during the maintenance window and we are now monitoring the performance of Playbooks. We will continue to update you through this medium.

5. monitoring Dec 02, 2021, 05:40 PM UTC

   Our team identified a problem with calls and have implemented a fix. We are currently monitoring the status of the service and will continue to update you through this medium.

6. investigating Dec 02, 2021, 10:48 PM UTC

   We are aware of slow-running queries on the database affecting the US Cluster, our team is developing a solution to deploy.

7. identified Dec 03, 2021, 08:30 AM UTC

   We have identified the issue and are implementing further mitigation and fix strategies.

8. identified Dec 03, 2021, 05:46 PM UTC

   We have implemented an automation to prevent long-running queries from blocking normal activities and are monitoring the system performance. We are aware that some users may still be experiencing a degree of latency on calls and this is part of our current focus for remediation.

9. monitoring Dec 05, 2021, 08:56 PM UTC

   A fix has been implemented for this issue and we are monitoring the performance of the system.

10. resolved Dec 08, 2021, 06:19 AM UTC

    This incident has been resolved.