KnowBe4 incident

Unexpected Emails being quarantined by PhishRIP

Minor Resolved View vendor source →

KnowBe4 experienced a minor incident on July 28, 2026 affecting PhishRIP, lasting 22h 57m. The incident has been resolved; the full update timeline is below.

Started
Jul 28, 2026, 08:00 PM UTC
Resolved
Jul 29, 2026, 06:58 PM UTC
Duration
22h 57m
Detected by Pingoru
Jul 28, 2026, 08:00 PM UTC

Affected components

PhishRIP

Update timeline

  1. investigating Jul 28, 2026, 08:00 PM UTC

    We are investigating an issue related to emails being quarantined by PhishRIP unexpectedly

  2. monitoring Jul 28, 2026, 08:35 PM UTC

    A fix has been implemented for this issue and PhishRIP queries are no longer quarantining messages unexpectedly. We are still investigating queries that may have been impacted

  3. resolved Jul 29, 2026, 06:58 PM UTC

    This incident has been resolved.

  4. postmortem Jul 30, 2026, 04:38 PM UTC

    ## **Summary** On July 28, 2026, an update to the PhishRIP service caused some PhishRIP search queries to match more broadly than they should have. Benign emails that didn't meet the configured sender criteria were quarantined across some customer accounts. Automated monitoring and customer support tickets surfaced the problem quickly. Engineering reverted the change within approximately 25 minutes of declaring the incident and stopped the unintended quarantining. Engineering then manually restored the incorrectly quarantined emails to customers’ inboxes. ## **Root cause** On July 28, 2026, we deployed a maintenance update to the query service behind PhishRIP. This update was designed to improve the handling of invalid sender addresses by allowing the system to fall back to searching only the domain portion of the address. However, existing validation logic used a regular expression that required an "@" symbol in the sender field. When a query used a bare domain, such as [example.com](http://example.com) with no username or "@" symbol, the regex flagged the domain as invalid. This conflict between the existing validation and the domain-search update caused the system to drop sender criteria entirely. The query ran against inboxes using only its remaining parameters, matching and quarantining emails well outside the intended sender. ## **Mitigation and remediation** **Code revert.** We weighed halting active PhishRIP operations mid-run against reverting the code. Halting the background tasks would have disrupted legitimate security workflows for every tenant, so reverting was the better option. The code revert was deployed and verified in production within approximately 40 minutes of escalation. **Email restoration.** To avoid releasing genuinely malicious emails back into customer environments, we manually restored them rather than in bulk. Engineering used internal admin tools to restore emails in batches, and worked with Support to verify each account ID, query ID, and receive customer approval first. All reported accounts were restored overnight. ## **Preventative measures** * **Regression coverage**: We are adding unit and integration tests for domain-only, non-standard, and bare-string sender queries to catch these input cases before deployment. * **Query guardrails**: We are adding a check in the query engine that blocks queries from running if their core filtering parameters are dropped during evaluation. ## **Conclusion** A single validation rule caused this: a regex that assumed every sender field contained an `@`. When queries used a bare domain, that assumption dropped the sender filter and the queries matched far more mail than intended. The fix was straightforward once we found it, and the revert stopped the damage within about 25 minutes of declaring the incident. Restoring quarantined mail was a harder process: we couldn't restore emails in bulk without risking genuinely malicious emails going back into inboxes. Our team chose to proceed carefully, working account by account with Support throughout the night. Two preventative measures would have caught this earlier: a test covering domain-only sender queries, and a guardrail that refuses to run a query when its sender filter has been stripped. Both are now planned for. The immediate issue is resolved, and all reported accounts have been restored. ## **Glossary** * **PhishRIP:** A PhishER feature that lets administrators search for and quarantine matching phishing emails across every user inbox in an organization. * **Regex \(regular expression\):** A pattern used in software to search text and validate input, such as checking that an email address is formatted correctly. * **Code revert:** Backing out a software update to return to the previous stable version.