Localytics experienced a minor incident on April 11, 2019 affecting Dashboard, lasting 1h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 11, 2019, 02:47 PM UTC
We have noticed sporadic instances of the Dashboard running slow or even failing to load. The team is currently investigating the root cause of this.
- identified Apr 11, 2019, 03:14 PM UTC
We've traced the root cause of this issue to our data warehouse and we are currently working on a solution.
- identified Apr 11, 2019, 03:27 PM UTC
We are working on switching over to our backup systems, but the process is taking longer than usual.
- monitoring Apr 11, 2019, 03:57 PM UTC
We have been fully switched over to our backup systems for about a half hour and Dashboard appears to be stable once again. We continue to closely monitor the situation.
- resolved Apr 11, 2019, 04:44 PM UTC
Dashboard continues to run smoothly and we are marking this issue as resolved. The root cause was a large number of queries hitting custom reports. Normally, this type of situation should not have any impact on the overall performance of the Dashboard, but a hidden setting on one of our systems caused all queries for the Dashboard to get stacked up behind these custom report queries, which caused the slowdown. We have changed this setting and will be implementing additional safeguards to prevent this situation from happening again. If you are still experiencing issues with Dashboard, please contact Support.