Amigo AI Outage History

Status: Monitoring Error rates have returned to nominal. We have also completed migration of all affected customer services to another LLM provider. We will continue monitoring for errors and provide an update in the next 30m. Affected components Conversations (Partial outage) Conversation (WebSockets) (Partial outage)

Status: Monitoring Problem: LLM outputs are occasionally returned as garbled text. Impact: Garbled LLM outputs are affecting organizations in the us-east-1 region, occurring in about 0.1% of messages. Causes: A hardware issue in us-east-1 combined with an ASCII to unicode prompt change that shifted tokenization boundaries. Steps to resolve: We are deploying a fix and expect to confirm its effectiveness within about 30m. Affected components Services (Degraded performance) Agents (Degraded performance) Conversations (Degraded performance) Conversation (WebSockets) (Degraded performance)

Status: Monitoring The mitigations that we deployed over the past day have resulted in significant reductions in the incidence of the garbled output across affected us-east-1 orgs. We have additionally rerouted the relevant traffic to another cloud provider region that we expect to completely resolve the issue whilst we work with our vendor to identify the root cause and resolution path. Affected components Conversation (WebSockets) (Degraded performance) Services (Degraded performance) Agents (Degraded performance) Conversations (Degraded performance)

Status: Monitoring We have cut over to other regions and are observing significantly reduced error rates and latency is returning to normal. We will continue to monitor for the next hour and provide an update. Affected components Conversations (Degraded performance)