Omni Analytics experienced a major incident on April 29, 2025 affecting EastUsa, lasting 5h 30m. The incident has been resolved; the full update timeline is below.
Affected components
EastUsa
Update timeline
- investigating Apr 29, 2025, 04:39 PM UTC
Users are experiencing slow responses in the US region, our team is investigating.
- monitoring Apr 29, 2025, 05:35 PM UTC
We have mitigated the issue and are monitoring the system.
- monitoring Apr 29, 2025, 07:57 PM UTC
We have identified the root cause of the issue and are implementing a fix.
- resolved Apr 29, 2025, 10:10 PM UTC
A model validation related process triggered repeated calls to an inefficient library function that exhausted memory in a subset of our US East infrastructure. A fix has been deployed that removes calls to the inefficient library function and we have validated it resolves the memory exhaustion.