Omni Analytics experienced a major incident on April 9, 2024 affecting EastUsa, lasting 6h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 09, 2024, 06:44 PM UTC
We are seeing elevated error rates. Investigating...
- investigating Apr 09, 2024, 06:45 PM UTC
We are continuing to investigate this issue.
- monitoring Apr 09, 2024, 07:10 PM UTC
The application is currently stable and we believe we have identified the cause.
- monitoring Apr 09, 2024, 08:00 PM UTC
From 11:39am to 11:52am Pacific Time the majority of queries run in Omni failed to complete. Our engineers have traced the issue back to a problematic set of queries against a specific database type that overflowed the memory available to our infrastructure, causing our querying system to crash repeatedly until the queries were cancelled. We have identified a fix that will be released today.
- resolved Apr 10, 2024, 01:04 AM UTC
This incident has been resolved.