Amplitude experienced a major incident on June 28, 2025 affecting Web Reporting and Evaluation API and 1 more component, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 28, 2025, 02:05 AM UTC
We're currently experiencing delays in our real-time ingestion and evaluation systems. This issue began around 5:00 PM PST. Our ingestion systems are operational but processing more slowly than normal. Customers using the HTTP endpoints in the US region are affected. Impacted users may not see events from the last 40–60 minutes reflected in charts. Event streaming out is also delayed. Our evaluation systems are operational but experiencing higher than normal latency. No data loss is expected. There is no impact to customers in the EU region. We're actively investigating and will share an update within the next hour, or sooner if possible.
- identified Jun 28, 2025, 03:01 AM UTC
We believe we've identified the root cause. We're implementing and deploying a hotfix to confirm this and restore normal data processing.
- identified Jun 28, 2025, 05:11 AM UTC
We’ve deployed the hotfix, and systems are now recovering. There is a backlog of several hours of data to process. We’re closely monitoring progress and adjusting configurations as needed. Thank you for your patience.
- identified Jun 28, 2025, 07:22 AM UTC
We are continuing to work on a fix for this issue.
- identified Jun 28, 2025, 07:55 AM UTC
We’re still looking into the issue to better understand and resolve it.
- identified Jun 28, 2025, 02:33 PM UTC
We have identified the bottleneck in our ingestion pipeline and applied a preliminary fix. Event ingestion via the HTTP endpoint is recovering. The real-time HTTP endpoint has fully caught up, while the batch endpoint is currently in the process of recovering. To prioritize real-time traffic, warehouse import has been paused. As a result, imports are currently delayed by up to 7 hours. We’re deploying another fix to improve performance and support full recovery. EU customers remain unaffected, and no data loss is expected.
- monitoring Jun 28, 2025, 10:50 PM UTC
We re-enabled warehouse imports for all customers at 12:30 PM PDT and are actively monitoring job lag recovery. We expect all jobs to catch up within the next 3 to 4 hours.
- monitoring Jun 29, 2025, 05:30 PM UTC
All warehouse import jobs have been fully caught up since 8am PDT, and our real-time ingestion and evaluation systems have been operating normally since yesterday. We are continuing to monitor the recovery of Data Export. Currently, the most delayed export jobs are behind by approximately 24 hours. We will keep you updated as progress continues. Thank you for your patience.
- monitoring Jun 29, 2025, 06:47 PM UTC
We are still monitoring the recovery of Data Export jobs. Currently, we expect to catch up on all Data Export jobs within the next four hours.
- resolved Jun 30, 2025, 12:27 AM UTC
All Data Export jobs have now caught up, and this incident has been resolved. No further action is required from customers, and we can confirm that no data was lost. Thank you for your patience throughout this process.