Omni Analytics experienced a critical incident on September 8, 2025 affecting EastUsa, lasting 1h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 08, 2025, 05:57 PM UTC
Our monitoring picked up a spike in errors and we have begun getting reports from users. The team is investigating on highest priority
- identified Sep 08, 2025, 06:02 PM UTC
We have identified the source of the issue and are in the process of rapidly reverting a change to resolve.
- identified Sep 08, 2025, 06:29 PM UTC
We have completed rolling out the mitigation for this incident and are now in the monitoring phase.
- resolved Sep 08, 2025, 07:15 PM UTC
We are now considering this incident resolved. A recent code change generated unexpectedly large database queries that saturated CPU resources, leading to query timeouts and subsequent connection failures. We initiated a rapid rollback once the cause was identified. Since mitigating we have reverted the change and are working to prevent similar issues in the future.