Omni Analytics Outage History

This incident has been resolved.

On June 15th, at approximately 18:19 PDT our monitoring alerted elevated load balancer response times due to a spike in traffic that we quickly identified as a DOS attack attempt. We implemented a firewall rule to block the attack 8 minutes later with response times returning to normal for most customers over the next 5 to 10 minutes. We aim to provide notice of significant disruptions lasting more than 10 minutes and are retroactively posting this to meet that goal.

Slack is reporting: "Slack services have largely returned to normal operational levels. Messaging, Workflows, Apps, Integrations, APIs, and Connections have all been restored." We are now marking this resolved regarding Omni integrations with slack.

The fix has fully deployed now seeing service return to normal What happened: A change we shipped to the MCP OAuth flow yesterday introduced a regression in how authentication tokens were handled by MCP tool calls. The OAuth handshake itself succeeded, so connecting to the MCP server worked, but invoking specific tools (like listing models or running queries) returned a “403 Invalid bearer token” error. Who this affected: Customers using MCP integrations that connect through callbacks.omniapp.co (Claude, ChatGPT, etc). Direct connections from CLI clients are not affected.

A scheduled update to an internal database table resulted in full page errors. A fix was put implemented to complete the update safely and the team is taking steps to ensure future updates do not cause errors.

The primary DNS provider reporting an issue has marked their incident resolved. We have not had any recent reports from Omni users.

The incident has been resolved, and all systems are back to operational status. We apologize for the disruption and are conducting a review to prevent the issue from happening again.

This incident has been resolved. The team has made changes to prevent the issue from happening again, and our monitoring shows healthy operations for the last 4 hours.

We've traced this incident to a configuration change on our end. We've reverted that change and we'll continue to monitor through future scaling events.

This incident has been resolved. Query processing has stabilized, all backlogs have cleared, and we are no longer seeing any errors or delays.

This incident has been resolved.

This incident is now resolved. In certain configurations combined with a very large number of model extensions it was possible to issue a query to the internal database that exhausted resources. Our mitigation is stable and a permanent fix is in development.

Our monitoring has not shown any signs of lingering issues in the hour following our cloud providers marking of their incident as closed. We are considering this incident resolved.

Due to an AWS outage we saw some logins and scheduled jobs failing. Things have since recovered and you can reattempt these actions.

A recent code change caused full page errors to be returned for users of advanced dbt and connection environment features. We initiated a fast rollback to mitigate and are working to prevent similar issues in the future.

This incident is now resolved.

We are now considering this incident resolved. A recent code change generated unexpectedly large database queries that saturated CPU resources, leading to query timeouts and subsequent connection failures. We initiated a rapid rollback once the cause was identified. Since mitigating we have reverted the change and are working to prevent similar issues in the future.

No errors in our monitoring have been detected since the mitigation fully rolled out. We are now considering this incident resolved.

We have isolated the source of the issue and reverted the change. We consider this incident resolved.

Relevant GCP services have all been marked healthy again on google's status page. Considering this incident resolved.

This incident has been resolved.

This incident has been resolved. A resource intensive process was initiated during a busy period that overwhelmed a subset of our internal resources and resulted in intermittent failure to complete requests. Manual scaling of component-specific resources was needed in addition to our automated scaling to recover. We have implemented some immediate efficiency improvements and are working on implementing additional larger improvements in the coming days.

A model validation related process triggered repeated calls to an inefficient library function that exhausted memory in a subset of our US East infrastructure. A fix has been deployed that removes calls to the inefficient library function and we have validated it resolves the memory exhaustion.

A fix has been implemented, and we’re monitoring to ensure the issue is resolved.

Slack has resolved this outage: https://slack-status.com/2025-02/1b757d1d0f444c34