Blits.ai incident

Some LLM / AI requests are falsy triggering guardrails

Blits.ai experienced a minor incident on May 13, 2025 affecting Blits EU Services (eu.platform.blits.ai) and Blits US Services (us.platform.blits.ai) and 1 more component, lasting 2h. The incident has been resolved; the full update timeline is below.

Started: May 13, 2025, 09:24 AM UTC
Resolved: May 13, 2025, 11:24 AM UTC
Duration: 2h
Detected by Pingoru: May 13, 2025, 09:24 AM UTC

Affected components

Blits EU Services (eu.platform.blits.ai)Blits US Services (us.platform.blits.ai)Blits UAE Services (uae.platform.blits.ai)

Update timeline

investigating May 13, 2025, 09:24 AM UTC

Some requests on production are triggering guardrails while it shouldn't. It seems to happen on a selection of the request. Root-cause under investigation.
resolved May 13, 2025, 11:24 AM UTC

A fix has been implemented. The issue was caused by a human error during the deployment of new guardrail settings for the system. As a result, chat history was inadvertently included in all guardrail checks. Since some bot responses contain JSON characters, the guardrails incorrectly flagged these as inappropriate, causing them to trigger far more frequently than intended. This problem affected a small subset of customers. It was dependent on a specific combination of the model in use and the output instructions applied. Because of this, the issue did not appear in our regression tests in the test environment.