Mixpanel experienced a minor incident on February 4, 2025, lasting 12h 40m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Feb 04, 2025, 08:20 AM UTC
Projects in the EU cluster are currently experiencing issues loading. Our engineering team is actively investigating a root cause. We appreciate your patience while our team works on this. If you have any questions, please contact support
- monitoring Feb 04, 2025, 08:24 AM UTC
Reports are back to loading normally. The engineering team is currently investigating the root cause.
- monitoring Feb 04, 2025, 12:02 PM UTC
Some customers may still experience latency issues when creating/loading reports. If so, please reach contact support and we'll look into your situation.
- investigating Feb 04, 2025, 12:48 PM UTC
Projects in the EU cluster continue experiencing issues loading. Our engineering team is actively investigating a root cause. We appreciate your patience while our team works on this.
- investigating Feb 04, 2025, 03:00 PM UTC
Projects in the EU cluster continue experiencing issues loading and our engineering team is actively investigating the root cause. We appreciate your patience while we work on this.
- monitoring Feb 04, 2025, 03:42 PM UTC
Our engineering team has deployed a fix and will continue monitoring its performance. We appreciate your patience throughout this situation. If you have any questions, please contact support.
- monitoring Feb 04, 2025, 05:21 PM UTC
We are continuing to monitor for any further issues.
- monitoring Feb 04, 2025, 07:11 PM UTC
For Projects in the EU Cluster, functionality to load reports should now be recovered. We are continuing to keep a close eye on any further issues.
- resolved Feb 04, 2025, 08:51 PM UTC
This incident has been resolved.
- postmortem Feb 08, 2025, 01:38 AM UTC
On February 4th, from approximately 7:56 am to 2:38 pm UTC, a temporary issue in our system caused a failure to load reports for projects hosted in our EU data center. During this period, all Mixpanel reports failed to load intermittently for these projects. Projects hosted in US and India were not affected. The disruption resulted from a code change that broke the proper queuing of background queries in our system. This overwhelmed our compute cluster, and queries from projects in the EU cluster failed to complete. Once we identified the issue, the change was quickly reverted and reports resumed loading normally. To mitigate future issues, we have added more alerts to our system to catch overload conditions sooner. Additionally, we are adding protection mechanisms to prevent this type of overload from reoccurring. We are committed to delivering accurate, timely analysis to support your critical decision-making, and we prioritize fast, high-quality customer support. If you have any questions or concerns, please feel free to reach out.