KnowBe4 Outage History

Started Nov 02, 2026, 02:10 PM UTC· resolved Nov 02, 2026, 07:24 PM UTC

1. monitoring Nov 02, 2026, 02:10 PM UTC

   We have identified an issue causing some delays with email processing for KnowBe4 Defend customers. Our team has implemented a fix and monitoring results. Delayed emails will automatically retry to process.

2. resolved Nov 02, 2026, 07:24 PM UTC

   This incident has been resolved.

3. postmortem Nov 05, 2026, 03:08 PM UTC

   From Thursday, October 29, 2026, at approximately 08:53 \(UTC\), to Saturday, October 31, 2026, at around 11:20 \(UTC\), some customers had inbound email rejected by Defend, and senders received a non-delivery report. This issue was caused by a software update to the Defend mail relay that applied stricter validation to long sender addresses. As a result, some valid inbound messages with long sender addresses, such as notifications and reports, were rejected. To resolve this issue, our team reverted the update. Defend mail flow returned to normal performance by about 11:20 \(UTC\) on October 31, 2026. We advise senders to resend any messages that were rejected during this period. On Monday, November 2, 2026, from approximately 13:40 to 13:45 \(UTC\), a follow-up update for the initial issue contained an error in its message processing, leading to temporary delays of up to 15 minutes for inbound email. Our team quickly detected the error and reverted the update. The sending mail servers retried affected messages automatically, so senders didn't need to resend them. To prevent this type of issue in the future, we're improving our monitoring and alerting for Defend mail relay errors so that we detect rejected email sooner. No data loss occurred as a result of these issues.

Read the full incident report →

Started Oct 31, 2026, 11:22 AM UTC· resolved Nov 02, 2026, 12:56 PM UTC

1. monitoring Oct 31, 2026, 11:22 AM UTC

   We have identified an issue that is causing some inbound emails for KnowBe4 Defend customers using Exchange Online connectors to be rejected. We have implemented a fix and are monitoring.

2. resolved Nov 02, 2026, 12:56 PM UTC

   This incident has been resolved.

3. postmortem Nov 05, 2026, 03:07 PM UTC

   From Thursday, October 29, 2026, at approximately 08:53 \(UTC\), to Saturday, October 31, 2026, at around 11:20 \(UTC\), some customers had inbound email rejected by Defend, and senders received a non-delivery report. This issue was caused by a software update to the Defend mail relay that applied stricter validation to long sender addresses. As a result, some valid inbound messages with long sender addresses, such as notifications and reports, were rejected. To resolve this issue, our team reverted the update. Defend mail flow returned to normal performance by about 11:20 \(UTC\) on October 31, 2026. We advise senders to resend any messages that were rejected during this period. On Monday, November 2, 2026, from approximately 13:40 to 13:45 \(UTC\), a follow-up update for the initial issue contained an error in its message processing, leading to temporary delays of up to 15 minutes for inbound email. Our team quickly detected the error and reverted the update. The sending mail servers retried affected messages automatically, so senders didn't need to resend them. To prevent this type of issue in the future, we're improving our monitoring and alerting for Defend mail relay errors so that we detect rejected email sooner. No data loss occurred as a result of these issues.

Read the full incident report →

Started Oct 20, 2026, 11:37 AM UTC· resolved Oct 23, 2026, 02:55 PM UTC

1. identified Oct 20, 2026, 11:37 AM UTC

   We've identified the cause of domain registration being unable to give advice and are working on implementing a fix. We'll continue to post on our status page with any new information or updates.

2. monitoring Oct 21, 2026, 03:33 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 23, 2026, 02:55 PM UTC

   This incident has been resolved.

4. postmortem Nov 03, 2026, 04:55 PM UTC

   From Saturday, July 18, 2026, to Friday, October 23, 2026, some customers using the New Domain Registration Advice features in Prevent and Defend received empty or incorrect data for many domains. This issue was caused by a defect at a third-party domain registration data provider. The provider stopped refreshing its records on July 18, 2026, and left blank creation dates for newly registered domains. Many older records also became outdated. As a result, the New Domain Registration Advice features in both products had no data or incorrect data for those domains. To resolve this issue, our team alerted and worked with the provider to correct the defect and restore missing records. Prevent and Defend returned to normal performance by 14:55 \(UTC\) on October 23, 2026. To prevent this type of issue in the future, we’ve added monitoring that alerts us to empty domain registration responses and outdated domain records. To make these features more resilient, we’re also evaluating additional sources for domain registration information. No data loss occurred as a result of this issue.

Read the full incident report →

Started Oct 13, 2026, 03:03 PM UTC· resolved Oct 13, 2026, 04:15 PM UTC

1. investigating Oct 13, 2026, 03:03 PM UTC

   We are currently investigating an issue affecting console login. We will provide more updates here when available.

2. monitoring Oct 13, 2026, 03:07 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 13, 2026, 04:15 PM UTC

   This incident has been resolved.

4. postmortem Oct 14, 2026, 02:29 PM UTC

   ## **Summary** On October 13, 2026, from approximately 14:19 to 15:05 \(UTC\), customers in all regions were unable to log in to the KnowBe4 Security Awareness Training \(SAT\) console, the KnowBe4 Learner Experience, and the PhishER console through KnowBe4 SAT. SAML and password logins returned errors, and the KnowBe4 SAT console didn't appear in the KnowBe4 platform. Some background processes, including user synchronization and training emails, also failed during this time. The incident was caused by a production release that conflicted with another change in our code. We reverted the release, and logins recovered across regions between 14:59 and 15:05 \(UTC\). No data loss occurred as a result of this issue. ## **Root Cause** A code deployment removed a startup configuration value to speed up application startup time. A separate component, added to our code after the original change was tested, didn’t account for that changed value. As a result, every attempt to save a user record raised an error, including the update that records sign-in attempts. ## **Customer Impact** Customers across all regions couldn't log in to the KSAT console via SAML or password. They were also unable to access the Learner Experience, and couldn’t access the PhishER console through the KnowBe4 SAT console. New user creation, user synchronization, and training emails failed during this time but were automatically retried after the deployment was reverted. ## **Mitigation and Remediation** We found that the deployment removed a configuration value that another component still required. This was mitigated by reverting the release, and we confirmed recovery across all regions through direct checks and automated tests. We also reverted the original change in our source code and prepared a corrected version that includes a compatibility fix, so the original improvement can be reintroduced safely. ## **Preventative Measures** We're adding additional code verifications and testing for deployments.

Read the full incident report →

Started Oct 13, 2026, 08:47 AM UTC· resolved Oct 13, 2026, 01:41 PM UTC

1. investigating Oct 13, 2026, 08:47 AM UTC

   We have received reports that the KnowBe4 Email Security Add-in is showing the error "The add-in KnowBe4 email security is preventing this message from being sent" when sending emails. We are investigating this issue and will update this page when we have more information.

2. monitoring Oct 13, 2026, 11:07 AM UTC

   We’ve implemented a fix for the Web Add-in sending errors and we’re monitoring the results to make sure no further issues occur.

3. resolved Oct 13, 2026, 01:41 PM UTC

   This incident has been resolved.

4. postmortem Oct 20, 2026, 12:48 PM UTC

   From Monday, October 12, 2026, at approximately 14:25 \(UTC\), to Tuesday, October 13, 2026, at around 15:07 \(UTC\), some customers in EU instances of Prevent and Protect had difficulty sending emails from Microsoft Outlook. Affected messages weren't sent, and the KnowBe4 Email Security add-in displayed the following error: "The add-in KnowBe4 Email Security is preventing this message from being sent." This issue was caused by an error in how an add-in version was prepared for release, which produced two slightly different copies of the same version. When Outlook loads the add-in, it downloads a set of files that must match. In EU instances, Outlook sometimes received files from both copies, so the add-in couldn't finish loading, and the message wasn't sent. Some devices also saved the failed download, so the error continued on those devices until a fix was applied. To resolve this issue, our team corrected the EU add-in files so that every customer receives a matching set. We then released an updated version of the add-in to all regions, which cleared any saved errors. The KnowBe4 Email Security add-in returned to normal performance by about 15:07 \(UTC\) on October 13, 2026. To prevent this type of issue in the future, we're changing how add-in versions are prepared and released, so that each version always produces the same set of files and every customer receives the same version at the same time. We're also changing the add-in so that failed downloads of add-in files aren't saved on devices. Lastly, we're adding alerts for add-in loading errors to detect them sooner. No data loss occurred as a result of this issue.

Read the full incident report →

Started Sep 28, 2026, 09:10 PM UTC· resolved Oct 02, 2026, 10:32 AM UTC

1. investigating Sep 28, 2026, 09:10 PM UTC

   We are currently investigating an issue regarding degraded performance for Prevent and Protect on our US servers.

2. monitoring Sep 29, 2026, 01:44 PM UTC

   We have identified an issue with the US instances of Prevent and Protect that has resulted in degraded performance. A fix has been implemented, and we will continue to monitor this issue.

3. resolved Oct 02, 2026, 10:32 AM UTC

   This incident has been resolved.

4. postmortem Oct 06, 2026, 03:51 PM UTC

   ## **Summary** From Monday, September 28, 2026, at approximately 16:00 \(UTC\), to Wednesday, September 30, 2026, at around 12:35 \(UTC\), some customers experienced delayed mail flow and intermittent problems accessing US instances of Prevent and Protect. On September 28, our monitoring showed heavy network traffic, with spikes at the top of each hour. Some customers began experiencing intermittent problems accessing US instances of the Prevent and Protect consoles. Internal lookups that mail processing depends on started timing out, and messages were queued as a result. Customers reported the issue to our support team, and we posted a status page notice. Performance recovered later that day as US traffic decreased. On September 29, our engineers reviewed the previous day's data across the mail gateways, the internal services they depend on, and the network layer. Mail flow delays and console access issues returned briefly, then eased as network traffic declined, following the same pattern as September 28. On September 30, from approximately 12:19 to 12:35 \(UTC\), our team increased the capacity of the US network services while service continued. Their load dropped immediately after the change, and Prevent and Protect returned to normal performance by around 12:35 \(UTC\) on September 30, 2026. We monitored peak traffic over September 30 and October 1 and saw no recurrences. No data loss occurred as a result of this issue. ## **Root Cause** This issue was caused by a capacity limit in the network services that handle traffic for our US mail-processing services. During peak traffic, and especially during the hourly spikes, the network services couldn't process connections fast enough. This added delay to traffic passing through them and caused some connections to drop. That traffic included internal requests our mail gateways send to look up distribution list members, so the right policies are applied. These requests are subject to a strict time limit so that a slow lookup can't hold up mail processing indefinitely. When the network services delayed requests beyond that limit, they failed. As a result, mail processing slowed, and messages were queued. Two factors made the issue worse and harder to identify. First, some internal requests also used these network services and were affected by the capacity limit. Second, the network services began to degrade before their compute use reached levels that would normally signal a problem. Standard compute monitoring didn't catch the network saturation, so the cause took longer to identify. Finding a safe way to increase capacity without interrupting service also took time, which is why the fix was applied on September 30. ## **Mitigation and Remediation** We found that the US network services didn't have enough capacity to handle peak traffic. We've mitigated this by increasing their capacity in a way that kept service running throughout the change. After the capacity increase, the network services handled peak traffic without further issues. We also found that compute use alone wasn't a reliable sign of network saturation. We've expanded our monitoring to track network health indicators in addition to compute use. Finally, we found it was difficult to find where failing internal requests broke down. We've added logging that shows where internal requests fail and time out. ## **Customer Impact** During peak US traffic on September 28 and 29, some customers experienced delays in processing and delivering inbound and outbound mail through Prevent and Protect. Messages addressed to distribution lists may have been processed more slowly while lookups were retried. Some customers also had intermittent difficulty logging in to or using the US console. The impact eased each day as US traffic decreased, and queued mail was processed once capacity recovered. After the change on September 30, we saw no further degradation during peak traffic. ## **Preventative Measures** To prevent this type of issue in the future, we've increased the capacity of the US network services and expanded our monitoring to track network health indicators in addition to compute use. We've also added logging to improve visibility into failed internal requests.

Read the full incident report →

Started Sep 24, 2026, 05:53 PM UTC· resolved Sep 25, 2026, 09:16 AM UTC

1. investigating Sep 24, 2026, 05:53 PM UTC

   We have identified an issue with Protect that has resulted in degraded performance, and our engineering team is looking into a fix.

2. monitoring Sep 24, 2026, 06:39 PM UTC

   We have identified an issue with Protect that has resulted in degraded performance. A fix has been implemented, and we will continue to monitor this issue.

3. investigating Sep 24, 2026, 07:57 PM UTC

   We have identified an issue with Protect & Prevent that has resulted in degraded performance, and our engineering team is looking into a fix.

4. monitoring Sep 24, 2026, 09:46 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Sep 25, 2026, 09:16 AM UTC

   This incident has been resolved.

6. postmortem Oct 06, 2026, 03:51 PM UTC

   ## **Summary** From Thursday, September 24, 2026, at approximately 17:07 \(UTC\), to Friday, September 25, 2026, at around 09:16 \(UTC\), some customers experienced intermittent slowness, errors, and delayed mail flow for US instances of Protect and Prevent. On September 24, an unusually large spike in email traffic placed sustained load on the network services that handle traffic for our US mail processing. Our monitoring alerted us at approximately 17:07 \(UTC\). Some customers saw slow or intermittent access to Protect and occasional errors when loading pages or advanced options. Access was inconsistent, so some customers could use Protect while others saw errors. Mail processing through US instances slowed, and some messages were delayed or didn't appear in the moderation inbox when expected. Performance appeared to improve, so we moved our status page notice to Monitoring at 18:39 \(UTC\). The issue later recurred, and customers reported continued slowness to our support team. At 19:57 \(UTC\), we moved the notice back to Investigating and expanded it to also include Prevent as the impact became clearer. Our team monitored the affected systems while the elevated mail volume subsided. As the load on the network services returned to normal, mail processing recovered, and mail queues returned to normal overnight. We confirmed that Protect was loading normally and that test messages were being delivered again, and then continued monitoring. Protect and Prevent returned to normal performance by around 09:16 \(UTC\) on September 25, 2026. No data loss occurred as a result of this issue. ## **Root Cause** This issue was caused by an unusually large spike in email traffic. The increased volume placed a sustained load on the network services that handle traffic for our US mail processing. Those network services operate within fixed capacity limits, and their capacity couldn't be scaled up during the spike. Because they couldn't keep up with the surge, they slowed down, and some processing failed intermittently. As a result, a large volume of mail was queued in US mail processing, some pages and actions in Protect returned errors, and some messages were delayed or didn't appear in the moderation inbox when expected. Two factors made the issue harder to manage. First, because access failed intermittently, some checks succeeded while customers continued to see errors, making recovery harder to confirm. Second, our monitoring showed peak load on the network services more clearly than the current load, which made it harder to judge in real time whether the load was still elevated. ## **Mitigation and Remediation** We found that the network services handling US mail processing had reached their fixed capacity limits during the traffic spike, and their capacity couldn't be increased during the event. Our team monitored the affected systems while the elevated mail volume subsided and network service load returned to normal. Mail processing then recovered, and queued mail was processed. Because the issue had previously appeared to improve before recurring, we confirmed recovery in several ways before resolving the incident. We confirmed that Protect was loading normally, that test messages were being delivered through US mail processing, and that support requests were progressing normally. We also monitored overnight and received no new reports from customers on US instances. ## **Customer Impact** Some customers had slow or intermittent access to Protect, and occasionally saw errors when loading pages or advanced options. Mail processing through US instances was delayed for Protect and Prevent. In some cases, messages were delayed or didn't appear in the moderation inbox when expected. Customers whose mail is encrypted, decrypted, or moderated as it passes through our US mail processing were more likely to be affected. Most Protect customers using the add-in weren't affected by the mail processing delays. The Protect and Prevent consoles remained available, although some console actions were affected by the slower mail processing. ## **Preventative Measures** To prevent this type of issue in the future, we're improving our monitoring of unusual increases in email volume. We're also improving our monitoring to show the current load on our network services, not only peak values. In addition, we're evaluating ways to scale our network capacity more dynamically during sudden traffic spikes, and assessing longer-term improvements to the network infrastructure that handles US mail processing. We've also since increased the capacity of US network services in response to[ a later incident](https://status.knowbe4.com/incidents/9wc6rjm8g70v).

Read the full incident report →

Started Sep 23, 2026, 09:52 AM UTC· resolved Sep 23, 2026, 02:58 PM UTC

1. identified Sep 23, 2026, 09:52 AM UTC

   We have received reports of users receiving "Access Denied" error messages while trying to login. We have identified the issue and are working on implementing a solution and will update this page when we have more information.

2. monitoring Sep 23, 2026, 10:57 AM UTC

   The issue has been identified and a fix has been implemented and will update this page when we have more information.

3. resolved Sep 23, 2026, 02:58 PM UTC

   This incident has been successfully resolved and all systems are fully operational.

Read the full incident report →

Started Sep 23, 2026, 09:00 AM UTC· resolved Sep 23, 2026, 02:58 PM UTC

1. investigating Sep 23, 2026, 09:00 AM UTC

   We have received reports that some users are unable to successfully submit webform submissions. We are investigating this issue and will update this page when we have more information.

2. identified Sep 23, 2026, 10:47 AM UTC

   The issue has been identified and a fix is been implemented. An update with more information will follow accordingly.

3. monitoring Sep 23, 2026, 10:58 AM UTC

   The issue has been identified and a fix has been implemented and will update this page when we have more information.

4. resolved Sep 23, 2026, 02:58 PM UTC

   This incident has been successfully resolved and all systems are fully operational.

5. postmortem Oct 20, 2026, 03:53 PM UTC

   On Wednesday, September 23, 2026, from approximately 08:15 to 11:18 \(UTC\), some customers experienced slowness and failed form submissions in UK instances of Secure Webforms. This issue was caused by database performance limits in the UK production environment following a recent change. A database setting also caused some queries to run inefficiently. As a result, response times increased and some form submissions failed. To resolve this issue, our team temporarily paused a background synchronization task to reduce load, adjusted the database settings to improve query efficiency, and increased storage throughput. Secure Webforms returned to normal performance by 11:18 \(UTC\). To prevent this type of issue in the future, we’re adding additional monitoring for database bandwidth. We’re also improving the efficiency of the background synchronization task. No data loss occurred as a result of this issue.

Read the full incident report →

Started Sep 21, 2026, 09:32 AM UTC· resolved Sep 22, 2026, 08:27 AM UTC

1. investigating Sep 21, 2026, 09:32 AM UTC

   We have received reports that users are unable to send emails via Web Access (Secure Message Portal) in the UK region, with Error "503 Service Unavailable". We are investigating this issue and will update this page when we have more information.

2. monitoring Sep 21, 2026, 02:36 PM UTC

   We’ve implemented a fix and are monitoring the results.

3. monitoring Sep 21, 2026, 02:38 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Sep 22, 2026, 08:27 AM UTC

   This incident has been successfully resolved and all systems are fully operational.

5. postmortem Oct 08, 2026, 01:45 PM UTC

   On Monday, September 21, 2026, from approximately 08:55 until Tuesday, September 22, 2026, at approximately 14:30 \(UTC\), some users experienced errors and were unable to send emails via Web Access \(Secure Message Portal\) in the UK region. This incident was caused by a configuration error during a database update, which led to intermittent service disruption. To resolve this issue, we reverted the configuration, and Web Access returned to normal performance by 14:30 \(UTC\) on September 22, 2026. To prevent this type of issue in the future, we are reviewing and updating our database configuration. No data loss occurred as a result of this issue.

Read the full incident report →

Started Sep 18, 2026, 11:07 AM UTC· resolved Sep 18, 2026, 04:48 PM UTC

1. investigating Sep 18, 2026, 11:07 AM UTC

   We have received reports that users are unable to login to the DE and UK instances of KSAT. We are investigating this issue and will update this page when we have more information.

2. resolved Sep 18, 2026, 04:48 PM UTC

   This incident has been resolved.

3. postmortem Oct 08, 2026, 02:07 PM UTC

   ## **Summary** On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job. ## **What Happened** A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users. When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 \(UTC\), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 \(UTC\). The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 \(UTC\) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 \(UTC\) before a permanent fix was deployed at approximately 15:00 \(UTC\). The incident was fully resolved by 19:12 \(UTC\) on September 18, 2026. ## **Root Cause** An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window. ## **Findings and Mitigations** We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions. ## **Preventive Measures** We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks. ## **Customer Impact and Recovery** Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occurred the next morning and were cleared manually within minutes. Some admin actions, such as user management, were also affected. Partner logins were unaffected. The secondary regression lasted around 10 hours and affected all regions for users who had previously failed a login attempt. By the afternoon of September 18, full recovery was confirmed across all regions, and the status page was marked as resolved. No data loss occurred as a result of this issue. ## **Conclusion** A single unbounded background transaction, combined with an automatic retry mechanism, caused a login outage in the US region. The specific triggering job and its retry behavior have been fixed, and login no longer depends on the contended write path. A same-incident regression and two minor recurrences were each resolved within hours. Remaining preventive work, including worker coordination and database transaction monitoring, is tracked above.

Read the full incident report →

Started Sep 17, 2026, 10:44 PM UTC· resolved Sep 18, 2026, 04:47 PM UTC

1. investigating Sep 17, 2026, 10:44 PM UTC

   We have received reports that users are unable to login to the US instance of KSAT. We are investigating this issue and will update this page when we have more information.

2. monitoring Sep 18, 2026, 01:28 AM UTC

   We’ve implemented a fix and we’re monitoring the results to make sure no further issues occur.

3. investigating Sep 18, 2026, 02:02 PM UTC

   We have received new reports of users being unable to login to the US instance of KSAT. We are investigating this issue and will update this page when we have more information.

4. resolved Sep 18, 2026, 04:47 PM UTC

   This incident has been resolved.

5. postmortem Oct 05, 2026, 09:16 PM UTC

   ## **Summary** On September 17, 2026, a background job archived a large user group and ran as a single unbounded database transaction. This transaction locked a large number of user rows, causing US region logins to time out because they write to the same table. After hitting a platform execution limit, the job was terminated and automatically retried five times in 15-minute intervals, resulting in a 75-minute outage. To resolve the issue, we deployed a fix that decoupled logins from the lock. We then deployed a follow-up fix for a regression introduced by the login change, as well as for the archival job. ## **What Happened** A prior change moved the "archive all users" action from a synchronous, time-out web request into an unbounded background job. When a large user group was archived, this triggered the new background job. The job locked all affected user rows in a single transaction, performing per-user writes and a synchronous external service call before committing. Login attempts, which write to the same table, were queued behind the locked job and timed out for US users. When the job reached a 15-minute platform execution limit, it was terminated mid-transaction, the transaction rolled back, and the job was redelivered to the queue. This cycle occurred five times, resulting in a 75-minute outage. When failures were detected internally at 22:35 \(UTC\), an incident was declared, and an engineer was paged to respond. The mitigation to decouple the write action from the login flow was deployed at approximately 23:44 \(UTC\). The mitigation itself caused a regression: any user in any region who had previously mistyped a password could no longer log in because resets for failed login attempts tried to write to a read-only database. By 09:17 \(UTC\) on September 18, 2026, the fix for this regression was deployed across regions. The original US-only archive job issue recurred twice more between 13:45 and 14:05 \(UTC\) before a permanent fix was deployed at approximately 15:00 \(UTC\). The incident was fully resolved by 19:12 \(UTC\) on September 18, 2026. ## **Root Cause** An unbounded, single-transaction background job for large-group user archival held exclusive locks on many affected rows, blocking the login process's own row write. A prior change to move this operation off a web request timeout and into a background job removed the only remaining limit on its duration. The platform's 15-minute execution limit then triggered automatic kill-and-redeliver cycles, repeating the lock contention five times and resulting in a 75-minute outage window. ## **Findings and Mitigations** We found that the unbounded archive transaction locked a large number of affected user rows, blocking other table writes. This was mitigated in the short term by decoupling the login process's write dependency from the affected table, and then fixed in the long term by correcting the archive job. We also found that uncoordinated worker concurrency amplifies lock contention. This was mitigated in the short term by manually reducing worker concurrency. The long-term fix to stagger and serialize these jobs is planned. Finally, we found that the login-fix regression caused all-region login failures for users with prior failed attempts because writes were rejected by a read-only database. This was fixed via a targeted code change that was deployed and verified across all regions. ## **Preventive Measures** We’re improving our large-group archival operations by bounding them, processing them in small, committed batches, and keeping external calls outside the database transaction. We’re also adding a change-check to a specific background worker to skip writes that don't change anything. Finally, we’re adding direct alerts for database transaction age and purging delays, in addition to periodic indirect checks. ## **Customer Impact and Recovery** Login failures lasted approximately 75 minutes in the US region. Two brief, US-only recurrences occurred the next morning and were cleared manually within minutes. Some admin actions, such as user management, were also affected. Partner logins were unaffected. The secondary regression lasted around 10 hours and affected all regions for users who had previously failed a login attempt. By the afternoon of September 18, full recovery was confirmed across all regions, and the status page was marked as resolved. No data loss occurred as a result of this issue. ## **Conclusion** A single unbounded background transaction, combined with an automatic retry mechanism, caused a login outage in the US region. The specific triggering job and its retry behavior have been fixed, and login no longer depends on the contended write path. A same-incident regression and two minor recurrences were each resolved within hours. Remaining preventive work, including worker coordination and database transaction monitoring, is tracked above.

Read the full incident report →

Started Sep 16, 2026, 05:47 PM UTC· resolved Sep 16, 2026, 08:58 PM UTC

1. identified Sep 16, 2026, 05:47 PM UTC

   We have identified an issue causing login and access issues for accounts on our UK instance.

2. monitoring Sep 16, 2026, 06:16 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Sep 16, 2026, 08:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 08, 2026, 07:00 AM UTC· resolved Sep 08, 2026, 07:00 AM UTC

1. resolved Sep 08, 2026, 07:00 AM UTC

   On Septermber 8th we made aware that inbound HTML emails rendered partially or blank in Classic Outlook due to a routine HTML library upgrade in Defend's harmful-code-removal engine interacting with Classic Outlook's legacy Word rendering engine. New Outlook and Outlook on the web were unaffected. This was strictly a display issue: no email was delayed, lost, or compromised, and threat detection remained fully active. Engineering has reverted the update to restore normal rendering. If you temporarily disabled harmful-code removal during this window, please re-enable it to ensure complete protection.

Read the full incident report →

Started Sep 04, 2026, 10:53 AM UTC· resolved Sep 07, 2026, 08:25 AM UTC

1. investigating Sep 04, 2026, 10:53 AM UTC

   We have received reports of an issue with delivery of Email verification MFA for Workspace. We are investigating this issue and will update this page when we have more information.

2. monitoring Sep 04, 2026, 02:24 PM UTC

   We have received reports that the issue with delivery of Email verification MFA for Workspace has been mitigated by Microsoft, and we are monitoring the results.

3. resolved Sep 07, 2026, 08:25 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 26, 2026, 04:19 PM UTC· resolved Aug 26, 2026, 07:51 PM UTC

1. investigating Aug 26, 2026, 04:19 PM UTC

   We have identified an issue where users may experience delays in messages being processed, and our engineering team is looking into a fix.

2. monitoring Aug 26, 2026, 04:46 PM UTC

   Our engineering team has implemented a fix for the issue that causing message processing delays in PhishER, and we are monitoring the results.

3. resolved Aug 26, 2026, 07:51 PM UTC

   Our engineering team has implemented a fix for the issue that causing message processing delays in PhishER.

4. postmortem Oct 16, 2026, 08:36 PM UTC

   ## **Summary** During a routine deployment on August 26, 2026, an infrastructure management framework unexpectedly removed both the primary message-processing queues and the secondary queues that hold messages that could not be processed. This resulted in a temporary interruption of incoming email delivery to PhishER customers' inboxes. No data loss occurred as a result of this issue. ## **Root Cause** On August 24, 2026, an internal configuration change left live configuration records out of sync with our code. A later code update did not revert those records. When a routine production deployment ran on August 26, 2026, the active configuration code did not contain the relocated resource records. As a result, the deployment treated the active queues as unmanaged resources and removed them. ## **Mitigation and Remediation** Automated monitoring alerted us to a drop in message processing, and an incident response team was assembled. We found that the queues had been deleted. This was mitigated in the short term by rerunning the affected production jobs to recreate the missing queues. We then verified that real-time email processing had resumed. Finally, we replayed email payloads held in cloud storage during the outage into customers' inboxes. ## **Timeline** All times are in UTC. * **August 24, 2026, 18:41:** An internal configuration change wrote updated resource records into live production configuration. * **August 26, 2026, 15:15:** A production deployment removed the inbound queues and the queues that hold messages that could not be processed in production, halting email delivery into customers' inboxes. * **August 26, 2026, 15:59:** Our automated monitoring alerted the engineering team to zero messages being processed, and an incident response team was paged. * **August 26, 2026, 16:01:** We confirmed the queues had been deleted and verified that incoming email payloads were still stored safely. * **August 26, 2026, 16:15:** We identified the configuration mismatch as the primary root cause. * **August 26, 2026, 16:25:** Our team reran the affected production jobs, and the queues were recreated. * **August 26, 2026, 18:44:** Our automated replay finished delivering all backlogged emails to customers' inboxes. ## **Preventative Measures** To prevent this type of issue in the future, we've removed a live-configuration step from pre-merge checks and separated deployment planning from production changes. We've also added automated checks to confirm queue health after releases are complete. ## **Customer Impact** From 15:15 to 16:25 \(UTC\) on August 26, 2026, incoming emails submitted to PhishER were not processed into customer inboxes. Inbound email payloads were stored safely in cloud storage throughout the disruption. After the queues were restored, an automated replay delivered held emails to customers' inboxes. All backlogged messages were delivered by 18:44 \(UTC\). No customer email data was lost as a result of this issue.

Read the full incident report →

Started Aug 21, 2026, 08:52 AM UTC· resolved Aug 21, 2026, 02:57 PM UTC

1. investigating Aug 21, 2026, 08:52 AM UTC

   We have received reports that the Gmail Phish Alert Button shows the error "[reportPhishAttempt()] An internal error has occurred. Please try again." when reporting phish. We are investigating this issue and will update this page when we have more information.

2. identified Aug 21, 2026, 12:03 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 21, 2026, 01:08 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 21, 2026, 02:57 PM UTC

   This incident has been resolved.

5. postmortem Aug 26, 2026, 03:55 PM UTC

   On Friday, August 21, 2026, from approximately 05:20 to 12:55 \(UTC\), some customers experienced intermittent failures when using the Gmail Phish Alert Button add-in to report suspicious emails, receiving the "An internal error has occurred" message instead of a successful report confirmation. The issue affected customers across all regions, with the highest error volumes concentrated in the EU-West and US-East service regions. This issue was caused by a disruption in a caching service on Google's Apps Script platform, which the add-in relies on to store per-user session configuration. The service began intermittently returning empty results on read while still accepting new writes, causing affected report submissions to fail. KnowBe4 engaged Google directly and filed a formal bug report between approximately 12:19 and 12:39 \(UTC\). Error volume declined sharply shortly after as the vendor-side condition cleared, and the Gmail Phish Alert Button returned to normal performance by approximately 12:55 \(UTC\). To prevent this type of issue in the future, we are continuing to work with Google to track the resolution of the underlying caching defect. No data loss occurred as a result of this issue.

Read the full incident report →

Started Aug 18, 2026, 02:29 PM UTC· resolved Aug 18, 2026, 08:14 PM UTC

1. investigating Aug 18, 2026, 02:29 PM UTC

   We are investigating an issue causing a Non-Delivery Report (NDR) to be returned when a message is reported via the Phish Alert Button and "Send Us a Copy" is enabled in Account Settings

2. monitoring Aug 18, 2026, 03:26 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 18, 2026, 08:14 PM UTC

   This incident has been resolved.

4. postmortem Sep 30, 2026, 08:06 PM UTC

   On Tuesday, August 18, 2026, from approximately 14:10 to 14:55 \(UTC\), customers using the Phish Alert Button \(PAB\) with the **Send Us a Copy** feature enabled received non-delivery reports for the **Send Us a Copy** mailbox. This issue was caused by a configuration change to the shared email routing infrastructure. This change renamed a messaging component used by both our staging and production environments. Because a production rule depended on this shared component, the rename caused delivery failures to the **Send Us a Copy** mailbox and triggered non-delivery reports. To resolve this issue, our team updated the affected configuration to reference the correct component, and the **Send Us a Copy** feature returned to normal performance by 14:55 \(UTC\). To prevent this type of issue in the future, we are separating email routing configurations for our staging and production environments. We are also reconciling legacy configurations into our current infrastructure management standards. No data loss occurred as a result of this issue.

Read the full incident report →

Started Aug 05, 2026, 07:29 PM UTC· resolved Aug 06, 2026, 09:56 PM UTC

1. investigating Aug 05, 2026, 07:29 PM UTC

   We are currently investigating an issue affecting Google PhishRIP queries, which may cause them to fail.

2. monitoring Aug 05, 2026, 09:24 PM UTC

   Based on our investigation into the recent errors affecting Google PhishRIP queries, it appears these may be related to suspended Gmail accounts. We are continuing to monitor the issue.

3. resolved Aug 06, 2026, 09:56 PM UTC

   Upon further investigation we have determined that the errors affecting Google PhishRIP queries were related to individual issues on Google accounts.

Read the full incident report →

Started Aug 04, 2026, 07:19 PM UTC· resolved Aug 05, 2026, 12:17 PM UTC

1. identified Aug 04, 2026, 07:19 PM UTC

   We have identified an issue preventing account settings from loading in the SAT console.

2. monitoring Aug 04, 2026, 07:51 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Aug 05, 2026, 12:17 PM UTC

   This incident has been resolved.

4. postmortem Sep 28, 2026, 02:35 PM UTC

   On Tuesday, August 4, 2026, from approximately 18:22 to 19:43 \(UTC\), some customers experienced issues loading the **Account Settings** page in the KnowBe4 console across the CA, DE, EU, UK, and US regions. This issue was caused by a series of application changes that increased the complexity of a GraphQL request used by the Account Settings experience. Each change remained below the platform's configured complexity threshold when evaluated individually, but combined, the request exceeded that threshold and prevented the page from loading. Engineering identified the complexity threshold as the contributing factor and deployed a corrective change beginning at approximately 19:14 \(UTC\). After the fix was verified across the affected regions, the KnowBe4 console returned to normal performance by approximately 19:43 \(UTC\). To prevent this issue in the future, we have increased the complexity threshold. No data loss occurred as a result of this issue.

Read the full incident report →

Started Jul 28, 2026, 08:00 PM UTC· resolved Jul 29, 2026, 06:58 PM UTC

1. investigating Jul 28, 2026, 08:00 PM UTC

   We are investigating an issue related to emails being quarantined by PhishRIP unexpectedly

2. monitoring Jul 28, 2026, 08:35 PM UTC

   A fix has been implemented for this issue and PhishRIP queries are no longer quarantining messages unexpectedly. We are still investigating queries that may have been impacted

3. resolved Jul 29, 2026, 06:58 PM UTC

   This incident has been resolved.

4. postmortem Jul 30, 2026, 04:38 PM UTC

   ## **Summary** On July 28, 2026, an update to the PhishRIP service caused some PhishRIP search queries to match more broadly than they should have. Benign emails that didn't meet the configured sender criteria were quarantined across some customer accounts. Automated monitoring and customer support tickets surfaced the problem quickly. Engineering reverted the change within approximately 25 minutes of declaring the incident and stopped the unintended quarantining. Engineering then manually restored the incorrectly quarantined emails to customers’ inboxes. ## **Root cause** On July 28, 2026, we deployed a maintenance update to the query service behind PhishRIP. This update was designed to improve the handling of invalid sender addresses by allowing the system to fall back to searching only the domain portion of the address. However, existing validation logic used a regular expression that required an "@" symbol in the sender field. When a query used a bare domain, such as [example.com](http://example.com) with no username or "@" symbol, the regex flagged the domain as invalid. This conflict between the existing validation and the domain-search update caused the system to drop sender criteria entirely. The query ran against inboxes using only its remaining parameters, matching and quarantining emails well outside the intended sender. ## **Timeline** All times are in UTC. | **Time** | **Event** | | --- | --- | | 19:36 | Our automated monitoring detected anomalous query execution patterns and alerted our engineers, who immediately began investigating. | | 19:51 | The issue was escalated to a high-severity production incident, and a response team was assembled. | | 19:52 | Our support team escalated the first customer tickets that reported unexpected quarantines, corroborating the monitoring alert. | | 19:54 | The root cause was traced to a recent queries API deployment, and engineering began reverting the change. | | 20:00 | Incident communications were posted to our status page, with PhishRIP marked as Degraded. | | 20:16 | The deployment was fully reverted. | | 20:35 | Production checks confirmed that subject and domain queries were again matching only their intended targets, and that no further messages were being quarantined in error. Our status page was updated accordingly. | | 20:39 | Engineering began restoring the incorrectly quarantined emails. | | 03:49 \(next day\) | Restorations were completed for all customer query sets logged through our support team. | ## **Mitigation and remediation** **Code revert.** We weighed halting active PhishRIP operations mid-run against reverting the code. Halting the background tasks would have disrupted legitimate security workflows for every tenant, so reverting was the better option. The code revert was deployed and verified in production within approximately 40 minutes of escalation. **Email restoration.** To avoid releasing genuinely malicious emails back into customer environments, we manually restored them rather than in bulk. Engineering used internal admin tools to restore emails in batches, and worked with Support to verify each account ID, query ID, and receive customer approval first. All reported accounts were restored overnight. ## **Preventative measures** * **Regression coverage**: We are adding unit and integration tests for domain-only, non-standard, and bare-string sender queries to catch these input cases before deployment. * **Query guardrails**: We are adding a check in the query engine that blocks queries from running if their core filtering parameters are dropped during evaluation. ## **Conclusion** A single validation rule caused this: a regex that assumed every sender field contained an `@`. When queries used a bare domain, that assumption dropped the sender filter and the queries matched far more mail than intended. The fix was straightforward once we found it, and the revert stopped the damage within about 25 minutes of declaring the incident. Restoring quarantined mail was a harder process: we couldn't restore emails in bulk without risking genuinely malicious emails going back into inboxes. Our team chose to proceed carefully, working account by account with Support throughout the night. Two preventative measures would have caught this earlier: a test covering domain-only sender queries, and a guardrail that refuses to run a query when its sender filter has been stripped. Both are now planned for. The immediate issue is resolved, and all reported accounts have been restored. ## **Glossary** * **PhishRIP:** A PhishER feature that lets administrators search for and quarantine matching phishing emails across every user inbox in an organization. * **Regex \(regular expression\):** A pattern used in software to search text and validate input, such as checking that an email address is formatted correctly. * **Code revert:** Backing out a software update to return to the previous stable version.

Read the full incident report →

Started Jul 17, 2026, 01:18 PM UTC· resolved Sep 11, 2026, 03:04 PM UTC

1. investigating Jul 17, 2026, 01:18 PM UTC

   We are investigating users being unable to purchase Protect licenses on store.knowbe4.com. We will update this page when we have more information.

2. monitoring Jul 17, 2026, 06:55 PM UTC

   We’ve implemented a fix and we’re monitoring the results to make sure no further issues occur.

3. investigating Jul 20, 2026, 06:24 PM UTC

   We have received reports that customers are unable to purchase Protect licenses on store.knowbe4.com. We are investigating this issue and will update this page when we have more information.

4. monitoring Jul 29, 2026, 06:40 PM UTC

   We’ve implemented a fix for the online store and we’re monitoring the results to make sure no further issues occur.

5. monitoring Aug 17, 2026, 09:02 PM UTC

   We are continuing to monitor the implemented fix for the online store, to ensure that no further issues occur.

6. resolved Sep 11, 2026, 03:04 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 16, 2026, 07:48 PM UTC· resolved Jul 16, 2026, 09:20 PM UTC

1. investigating Jul 16, 2026, 07:48 PM UTC

   We have identified an issue where users may receive a redirect upon logging into Defend, and our engineering team is looking into a fix.

2. identified Jul 16, 2026, 08:09 PM UTC

   A fix has been identified, and we will be monitoring the results as it rolls out.

3. resolved Jul 16, 2026, 09:20 PM UTC

   This incident has been resolved.

4. postmortem Aug 20, 2026, 09:32 PM UTC

   On Thursday, July 16, 2026, from approximately 19:57 to 21:12 \(UTC\), some customers experienced difficulty logging in to the Defend console, seeing a **Find Out More** screen instead of console access. This issue was caused by a synchronization issue between Salesforce and our internal licensing systems, which incorrectly set some customer license counts to zero. As a result, affected customers were unable to log in. To resolve this issue, our team applied a temporary license override to restore access for affected customers while correcting the underlying licensing records in Salesforce. Once corrected, the systems automatically synchronized the accurate values, and the Defend console access returned to normal performance by approximately 21:12 \(UTC\). To prevent this type of issue in the future, we are implementing improvements to the Salesforce synchronization process to ensure accurate license counts. No data loss occurred as a result of this issue.

Read the full incident report →

Started Jul 15, 2026, 04:46 PM UTC· resolved Jul 16, 2026, 02:04 PM UTC

1. investigating Jul 15, 2026, 04:46 PM UTC

   We are investigating users being unable to purchase Protect licenses on store.knowbe4.com. We will update this page when we have more information.

2. investigating Jul 15, 2026, 04:48 PM UTC

   We are continuing to investigate this issue.

3. investigating Jul 15, 2026, 04:51 PM UTC

   We are investigating users being unable to purchase Protect licenses on store.knowbe4.com. We will update this page when we have more information.

4. investigating Jul 15, 2026, 04:54 PM UTC

   We are continuing to investigate this issue.

5. monitoring Jul 16, 2026, 01:23 AM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jul 16, 2026, 02:04 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 10, 2026, 02:30 PM UTC· resolved Jul 10, 2026, 02:30 PM UTC

1. resolved Jul 10, 2026, 02:30 PM UTC

   An upstream feature flag service issue degraded model routing in collaboration-inference, causing PhishML evaluations to return 0/0/0 default errors. The third party deployed a fix, and a code update was deployed to production adding static fallback routing. All services have recovered and are operating normally.

Read the full incident report →