CyberQP incident
Dashboard Slowness Navigating Accounts Inside Customer
CyberQP experienced a notice incident on May 2, 2024 affecting Web Dashboard and Web Dashboard, lasting 7h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 02, 2024, 03:18 PM UTC
We are aware of dashboard slowness and increased load times when navigating to accounts once clicked into a Customer. Our team is actively investigating and will update as more information is made available.
- monitoring May 02, 2024, 03:45 PM UTC
A fix has been put in place and dashboard response times are coming back to normal. We will continue to monitor the situation to ensure that the dashboard stays responsive. Another update will be posted when the work is completed.
- resolved May 02, 2024, 10:32 PM UTC
Dashboard, ConnectWise POD, and AutoTask Insights are all responsive and performing as expected. We will continue to monitor. If you are still having challenges with Dashboard speed, or functionality, please respond or create a new ticket to [email protected].
- postmortem May 08, 2024, 09:42 PM UTC
On May 1st, we deployed a standard maintenance release that included a change to unify where we pull data from in our backend services. On May 2nd, during peak usage hours, we received reports of certain pages loading too slowly. At the same time, we noticed abnormal load on one of our services causing it to take a long time to respond to requests. We began troubleshooting and determined it was related to the unifying code we deployed and decided to revert the change. We're taking steps to mitigate this in the future by adding additional monitoring and alerting as well as adding steps in our design and test processes to better understand and forecast performance impacts.