KnowBe4 Outage History

Started Feb 10, 2026, 02:05 PM UTC· resolved Feb 10, 2026, 03:52 PM UTC

1. investigating Feb 10, 2026, 02:05 PM UTC

   We are currently investigating an issue with PhishER integration delays in our EU instance.

2. monitoring Feb 10, 2026, 02:38 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Feb 10, 2026, 02:40 PM UTC

   A fix was identified and applied and we are continuing to monitor for any further issues.

4. resolved Feb 10, 2026, 03:52 PM UTC

   This incident has been resolved.

5. postmortem Feb 13, 2026, 10:46 PM UTC

   On Tuesday, February 10, 2026, from approximately 13:57 to 16:01 \(UTC\), some users experienced delays with the integration service on the EU instance of the PhishER console. This incident was caused by a system upgrade that unexpectedly affected the integration service, causing errors in the production environment. To resolve this issue, we reverted the update, and the PhishER console returned to normal performance by 16:01 \(UTC\). No data loss occurred as a result of this issue.

Read the full incident report →

Started Feb 10, 2026, 12:25 PM UTC· resolved Feb 11, 2026, 02:08 PM UTC

1. investigating Feb 10, 2026, 12:25 PM UTC

   We are seeing intermittent reports of file uploads failing to customers Workspace instances and are currently investigating the issue.

2. investigating Feb 10, 2026, 12:57 PM UTC

   We are seeing degraded performance across customer's Workspace instances. Some customers may see additional impact based on their current usage.

3. investigating Feb 10, 2026, 01:27 PM UTC

   We have identified this outage is impacting all instances and not just the UK instance as originally reported. We are continuing to investigating this issue and will update this page when we have more information

4. resolved Feb 11, 2026, 02:08 PM UTC

   This incident has been resolved.

5. postmortem Feb 25, 2026, 03:38 PM UTC

   On Tuesday, February 10, 2026, customers in the UK region began experiencing upload delays with Workspace at approximately 11:01 \(UTC\), due to instability within a messaging node. The node was isolated at 12:20 \(UTC\) to protect cluster integrity and was successfully reintegrated at 13:40 \(UTC\), restoring normal upload processing shortly after. Beginning at approximately 12:10 \(UTC\), a subset of Workspace instances across multiple regions became slow or temporarily unresponsive due to memory exhaustion on their underlying virtual machines. As each Workspace instance operates independently, the impact varied by customer. To resolve this issue, we identified and progressively redeployed the affected instances, starting at 14:37 \(UTC\), with the majority restored shortly after. The final affected instance was confirmed healthy at 19:27 \(UTC\). Our analysis confirmed that abnormal memory consumption by the Azure Monitoring Agent caused the host memory exhaustion observed on impacted instances. An infrastructure update scan occurred earlier that morning, at 08:22 \(UTC\). Current evidence indicates that this scan triggered an underlying defect in the monitoring agent. The scan itself was non-disruptive in design and was a platform security procedure, not intended as a customer-impacting maintenance activity. We confirmed that all systems were stable on February 10 and completed a deep review of data validation on February 11, 2026. No data loss or security compromise occurred. To prevent this type of impact in the future, we are strengthening monitoring safeguards around host memory behaviour, enhancing messaging cluster resilience, reinforcing update governance controls, and formalising rapid restoration procedures.

Read the full incident report →

Started Feb 04, 2026, 04:17 PM UTC· resolved Feb 04, 2026, 11:00 PM UTC

1. investigating Feb 04, 2026, 04:17 PM UTC

   We are currently investigating an issue with inbound email delivery delay

2. monitoring Feb 04, 2026, 04:34 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 04, 2026, 11:00 PM UTC

   This incident has been resolved.

4. postmortem Feb 06, 2026, 04:29 PM UTC

   On Wednesday, February 4, 2026, from approximately 15:30 to 16:00 \(UTC\), some US users experienced email delivery delays in the Defend product. This issue was caused by a brief networking fluctuation, resulting in email delivery delays of up to 10 minutes during the affected period. Once the cloud environment returned to healthy operating levels, Defend returned to normal performance, and all delayed emails were delivered by 16:00 \(UTC\). No data loss occurred as a result of this issue.

Read the full incident report →

Started Jan 27, 2026, 04:20 PM UTC· resolved Jan 27, 2026, 06:04 PM UTC

1. identified Jan 27, 2026, 04:20 PM UTC

   We've identified the cause of the Library not loading within the learner experience and are working on implementing a fix.

2. resolved Jan 27, 2026, 06:04 PM UTC

   This incident has been resolved.

3. postmortem Jan 30, 2026, 03:10 PM UTC

   On Tuesday, January 27, 2026, from approximately 14:46 \(UTC\) to 16:22 \(UTC\), some users experienced issues with loading the **Library** tab of the Learner Experience \(LX\). This incident was caused when an update to the **Library** tab contained a code change that conflicted with another recent update. The conflicting code changes unexpectedly caused an error on the LX that was not detected by our automated testing and monitoring until both updates were deployed to the production environment. To resolve this issue, we rolled back the updates, and the LX returned to normal performance by 16:22 \(UTC\). To prevent this type of issue in the future, we updated the code to prevent the error when the update is redeployed. We also improved the testing and monitoring protocols. No data loss occurred as a result of this issue.

Read the full incident report →

Started Jan 22, 2026, 08:15 PM UTC· resolved Jan 23, 2026, 12:27 PM UTC

1. identified Jan 22, 2026, 08:15 PM UTC

   We are aware of an ongoing incident with Microsoft 365 causing emails to not be received by Defend. This is resulting in emails not being delivered to inboxes. This outage is also resulting in the Phish Alert Button being unable to report emails. We are investigating this issue and will update this page when we have more information.

2. identified Jan 22, 2026, 08:36 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Jan 22, 2026, 10:25 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Jan 23, 2026, 08:24 AM UTC

   A fix has been implemented and we are monitoring the results.

5. monitoring Jan 23, 2026, 08:25 AM UTC

   Microsoft 365 services appear to have recovered. We are actively monitoring all dependent systems to ensure email flow and related functionality remain fully operational.

6. resolved Jan 23, 2026, 12:27 PM UTC

   This incident has been resolved.

7. postmortem Feb 02, 2026, 05:35 PM UTC

   On January 22, 2026, from approximately 19:30 \(UTC\) to January 23, 2026, at approximately 05:00 \(UTC\), a significant service disruption in Microsoft’s North American infrastructure triggered a series of service degradations across KnowBe4 products. This Microsoft service disruption primarily affected Exchange Online connectivity and the responsiveness of the Microsoft Graph API and the Office JS API. The incident created a two-fold impact on our environment. First, Microsoft’s SMTP relays began rejecting connections, which prevented our secure gateways from returning processed mail to customer environments. As a result, Defend and Protect began safely queuing messages and reached approximately 530,000 emails held in a secure retry state. Prevent also had partial disruptions in which moderation messages could not be transmitted. These disruptions led to delivery delays and triggered preconfigured default actions. While the SMTP relays were down, API instability also affected our suite of Microsoft Outlook add-ins. Both the Phish Alert Button \(PAB\) and the KnowBe4 Email Security add-in experienced functional failures and extreme latency because they could not retrieve necessary data from Microsoft’s backend to process reports and display security nudges. At the same time, the deployment center recorded failed installations for new customers, since the system-generated "handshake" that is required to verify transport rules was blocked by the same Microsoft relay failures. As Microsoft began restoring its North American services on January 23, 2026, at around 01:40 \(UTC\), our systems initiated automated recovery protocols. Using exponential backoff logic, our gateways began automatically clearing the mail queues. By 05:00 \(UTC\), the Defend and Protect services returned to full performance, queues were fully processed, and all deployment health checks for Defend and Prevent were confirmed successful. No data loss occurred as a result of this issue.

Read the full incident report →

Started Jan 21, 2026, 06:18 PM UTC· resolved Jan 22, 2026, 12:05 PM UTC

1. monitoring Jan 21, 2026, 06:18 PM UTC

   We've received reports of some customers having mail flow issues intermittently. We have identified and implemented a fix, however the fix may take 1-4 hours to propagate.

2. resolved Jan 22, 2026, 12:05 PM UTC

   This incident has been resolved.

3. postmortem Feb 04, 2026, 08:49 PM UTC

   On Wednesday, January 21, 2026, from approximately 14:59 to Thursday, January 22, 2026, at approximately 10:11 \(UTC\), some customers experienced delays with outbound email delivery for UK instances of Prevent, Defend, and Protect. This incident was caused by an external email reputation service provider that temporarily restricted one of our outbound email IP addresses. Some messages routed through this restricted address experienced temporary delays while we attempted delivery using other sending addresses. To resolve this issue, our team reached out to the service provider to have the IP address taken off their blocklist. Once the restriction was lifted at 17:33 \(UTC\), our systems automatically retried the remaining queued messages, and outbound email delivery returned to normal performance by January 22, 2026, at 10:11 \(UTC\). To prevent this type of issue in the future, we are investigating additional mitigations for external reputation factors. We are also enhancing our existing monitoring to identify and respond to outbound email reputation changes more quickly. No data loss occurred as a result of this issue.

Read the full incident report →

Started Dec 29, 2025, 10:37 AM UTC· resolved Dec 29, 2025, 12:10 PM UTC

1. identified Dec 29, 2025, 10:37 AM UTC

   We have identified high latency with webform submissions and are currently investigating the issue.

2. resolved Dec 29, 2025, 12:10 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 22, 2025, 04:20 PM UTC· resolved Dec 22, 2025, 05:04 PM UTC

1. investigating Dec 22, 2025, 04:20 PM UTC

   We are aware of an issue preventing users from logging in to the KnowBe4 Security Awareness Training (KSAT) console. We are investigating this issue and will update this page when we have more information.

2. monitoring Dec 22, 2025, 04:56 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 22, 2025, 05:04 PM UTC

   This incident has been resolved.

4. postmortem Jan 20, 2026, 05:24 PM UTC

   On Monday, December 22, 2025, from approximately 16:12 to 16:50 \(UTC\), some customers experienced errors when attempting to log in and log out across the KSAT console. This issue was caused by a cryptography library update required for a new feature. To resolve this issue, we reverted the update, and the KSAT console returned to normal performance by 16:50 \(UTC\). To prevent this type of issue in the future, we have updated our deployment workflows to require sanitation tests before new deployments are sent. No data loss occurred as a result of this issue.

Read the full incident report →

Started Dec 15, 2025, 06:54 PM UTC· resolved Dec 16, 2025, 01:59 PM UTC

1. investigating Dec 15, 2025, 06:54 PM UTC

   We are investigating an issue preventing user lists from being populated in KSAT. This is impacting the Users tab, user lists within Phishing campaigns and user groups. We'll continue to post on our status page with any new information or updates.

2. investigating Dec 15, 2025, 07:06 PM UTC

   We are continuing to investigate an issue preventing user lists from being populated in KSAT. This is impacting the Users tab, user lists within groups and within campaign subtabs. We'll continue to post on our status page with any new information or updates.

3. monitoring Dec 15, 2025, 07:55 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Dec 15, 2025, 07:58 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Dec 16, 2025, 01:59 PM UTC

   This incident has been resolved.

6. postmortem Jan 20, 2026, 07:10 PM UTC

   On Monday, December 15, 2025, from approximately 18:39 to 19:32 \(UTC\), customers were unable to load user data across all regions of the KSAT console. This issue was immediately identified by our monitoring systems, and the root cause was traced to a database migration during a product deployment. During the migration, a column was intentionally removed from the database as part of a feature deprecation task. However, due to a caching inconsistency between the application images and the database structure, a synchronization gap was created. Our schema cache system retained old references to the deleted column in the running application images. Because the old application image was still querying for the deleted column and the database no longer contained it, active instances began throwing errors. This issue resulted in a failure to load user data across all regions. Our engineering team restored service by re-adding the deleted column to both tables. This action resolved the database query inconsistency and allowed the system to load user data correctly. The KSAT console returned to normal performance by 19:32 \(UTC\). No data loss occurred as a result of this issue.

Read the full incident report →

Started Dec 15, 2025, 09:49 AM UTC· resolved Dec 15, 2025, 12:32 PM UTC

1. identified Dec 15, 2025, 09:49 AM UTC

   We are aware of an ongoing incident affecting Azure's Front Door service, which is intermittently impacting several of our products and features. We are investigating this issue and will update this page as soon as we have more information.

2. monitoring Dec 15, 2025, 10:24 AM UTC

   Services currently appear stable. We are continuing to monitor the situation closely and will update this incident if any further impact is observed.

3. resolved Dec 15, 2025, 12:32 PM UTC

   The Azure Front Door issue is no longer impacting services. This incident is now resolved.

Read the full incident report →

Started Dec 04, 2025, 05:04 PM UTC· resolved Dec 05, 2025, 12:49 PM UTC

1. identified Dec 04, 2025, 05:04 PM UTC

   We've identified why the Saved Reports tab is not loading and are working on implementing a fix. We'll continue to post on our status page with any new information or updates.

2. monitoring Dec 04, 2025, 08:58 PM UTC

   We’ve implemented a fix for the Saved Reports tab and we’re monitoring the results to make sure no further issues occur.

3. resolved Dec 05, 2025, 12:49 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 02, 2025, 05:50 PM UTC· resolved Dec 03, 2025, 01:14 PM UTC

1. investigating Dec 02, 2025, 05:50 PM UTC

   We are investigating an issue affecting the ability to login to KnowBe4 training via the login link in training notifications. Our team is working on this issue and we will update this page when we have more information.

2. monitoring Dec 02, 2025, 09:39 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 03, 2025, 01:14 PM UTC

   This incident has been resolved.

4. postmortem Jan 06, 2026, 04:59 PM UTC

   On December 2, 2025, from approximately 16:30 to 17:00 \(UTC\), some customers experienced issues with using login links in training notifications. This issue was caused by a code deployment for the KSAT login page, which generated incorrect login URLs. To resolve this issue, we reverted the deployment, and the KSAT console returned to normal performance by 17:00 \(UTC\). To prevent this type of issue in the future, we have improved our workflow for similar code deployments to include additional monitoring and testing. No data loss occurred as a result of this issue.

Read the full incident report →

Started Dec 02, 2025, 04:47 PM UTC· resolved Dec 03, 2025, 11:41 AM UTC

1. identified Dec 02, 2025, 04:47 PM UTC

   Some customers have reported issues with logging into the ESI page on the US instance. We have currently identified this issue.

2. identified Dec 02, 2025, 04:51 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Dec 02, 2025, 06:49 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 03, 2025, 11:41 AM UTC

   This incident has been resolved.

5. postmortem Feb 11, 2026, 07:08 PM UTC

   On Tuesday, December 2, 2025, from approximately 16:12 to 18:35 \(UTC\), some US users experienced issues when signing in to the Prevent, Defend, Protect, Secure Web Form, and Workspace products. This incident was caused by the recent deployment of a new monitoring agent. The agent consumed more resources than expected, leading to periodic resource strain and service instability. To resolve this issue, we optimized the agent's resource usage and increased the total system capacity, and the affected products returned to normal performance by 18:35 \(UTC\). To prevent this type of issue in the future, we are maintaining the increased system capacity for current and future projects. No data loss occurred as a result of this issue.

Read the full incident report →

Started Nov 24, 2025, 03:50 PM UTC· resolved Nov 24, 2025, 05:11 PM UTC

1. investigating Nov 24, 2025, 03:50 PM UTC

   We have received reports that the US instance of PhishER is not loading. We are investigating this issue and will update this page when we have more information.

2. monitoring Nov 24, 2025, 04:02 PM UTC

   We’ve implemented a fix and are monitoring the results to make sure no further issues occur.

3. resolved Nov 24, 2025, 05:11 PM UTC

   This incident has been resolved.

4. postmortem Dec 17, 2025, 05:16 PM UTC

   **Incident Summary** On November 24, 2025, PhishER experienced a service disruption affecting the ability of administrators in the US region to log in to the platform. The issue was identified as a database locking event caused by a deployment of a new column. The service was restored by terminating the locking process, and a code revert was subsequently performed to ensure stability. **Timeline of Events** Our monitoring and alerting systems notified engineers of a potential anomaly at approximately 15:30 UTC, prompting an immediate investigation. Shortly thereafter, we received reports that PhishER administrators in the US region were unable to log in. By 15:40 UTC, the engineering team identified the source of the issue: a database deployment job had stalled. Further investigation by engineers revealed that a specific operation to add a new column to the "accounts" table was causing a lock. The database team terminated the stuck process, and by 15:51 UTC, access to the PhishER console was fully restored, allowing users to log in successfully. Following the restoration of service, engineers confirmed that no other regions were impacted. A code revert was completed at approximately 16:30 UTC to prevent recurrence. **Detailed Timeline \(UTC\)** Time | Event Description ~15:30 | Monitoring and alerting systems detected a potential anomaly regarding login latency. 15:35 | Customer support received initial reports that PhishER administrators in the US region were unable to log in. 15:40 | The engineering team identified the source of the issue: a background database deployment job had stalled. 15:45 | Database engineers confirmed a specific operation on the "accounts" table was causing a lock, preventing user sessions. 15:51 | The database team terminated the stuck process. Access to the PhishER console was immediately restored. 16:00 | Engineers verified that service health had returned to normal levels and confirmed no other regions were impacted. ~16:30 | A code revert of the deployment was successfully deployed to prevent the issue from recurring. ‌ **Root Cause** The root cause of this incident was a database deployment script that attempted to add a new boolean column \(advanced\_mode\_enabled\) with a default value to the "accounts" table. This operation resulted in a database lock due to contention, preventing legitimate user sessions from being established during the deployment attempt. **Findings and Mitigations** The investigation concluded that the disruption was triggered by a database deployment script intended to add a boolean column \(advanced\_mode\_enabled\) with a default value to the "accounts" table. Because this table is used frequently, the alteration caused high contention, resulting in a database lock that queued and subsequently timed out legitimate user sessions. **Mitigations** * Immediate Action: The database team manually terminated the locked database query. This action immediately released the lock, allowing queued login requests to process and restoring service availability. * Stabilization: To ensure the system remained stable and to prevent the deployment from automatically retrying and causing further locking, a code revert was issued and deployed. **Preventive Measures** * Maintenance Window Scheduling: The specific database change required for this feature has been rescheduled to occur during a designated off-hours maintenance window, ensuring zero impact on active user traffic. **Conclusion** This incident was caused by database contention during a deployment, which temporarily impacted PhishER login availability in the US region only. Full service stability was restored by reverting the changes. We have implemented enhanced protocols for scheduling high-impact database operations during maintenance windows. These measures are designed to prevent recurrence and ensure the continued reliability of the platform. ‌ **Glossary of Technical Terms** * Database Lock: A safeguard mechanism that temporarily prevents multiple processes from accessing or modifying the same data simultaneously. In this incident, the lock unintentionally prevented the login service from reading user account data. * Code Revert: The process of undoing specific changes in the software codebase to return the system to a previously known stable state. * Boolean Column: A specific type of data field used to store True or False values. The deployment in question attempted to add a "flag" \(True/False\) to enable a new feature for accounts. * Maintenance Window: A pre-scheduled period of time designated for performing technical updates or changes, minimizing the risk of unplanned disruption to users.

Read the full incident report →

Started Nov 21, 2025, 12:30 PM UTC· resolved Nov 21, 2025, 12:50 PM UTC

1. monitoring Nov 21, 2025, 12:30 PM UTC

   Between 10:10am - 10:25am UTC on 21st of Nov, there was an issue with web pages in the console loading slowly or in some cases, not loading. This issue was quickly identified and resolved.

2. resolved Nov 21, 2025, 12:50 PM UTC

   This incident has been resolved.

3. postmortem Dec 16, 2025, 02:17 PM UTC

   On November 21, 2025, we deployed a configuration change to KSAT messaging. This deployment caused users to experience intermittent HTTP 500 errors from approximately 10:22 to 15:37 \(UTC\), and prevented users from logging in or interacting with the EU instance of the KSAT console. To resolve this issue, we rolled back the configuration change, and the KSAT console returned to normal performance by 15:37 \(UTC\). No data loss occurred as a result of this issue.

Read the full incident report →

Started Nov 14, 2025, 04:41 PM UTC· resolved Nov 14, 2025, 11:48 PM UTC

1. investigating Nov 14, 2025, 04:41 PM UTC

   We have received reports that users are unable to access training. We are investigating this issue and will update this page when we have more information.

2. investigating Nov 14, 2025, 04:58 PM UTC

   We have received additional reports that KSAT customer account creation is unavailable. We will continue to investigate this issue and will update this page when we have more information.

3. monitoring Nov 14, 2025, 05:10 PM UTC

   We’ve implemented a fix and users should be able to login at this time. We will continue to monitor the results to make sure no further issues occur.

4. resolved Nov 14, 2025, 11:48 PM UTC

   This incident has been resolved.

5. postmortem Jan 12, 2026, 02:35 PM UTC

   # **External Technical Root Cause Analysis - KSAT - Unable to access training** ‌ This report outlines the detailed findings and mitigations associated with a service interruption impacting the KnowBe4 Learner Experience \(LX\) platform. The interruption was caused by a schema mismatch between the backend database and the frontend application during a deployment intended to remove deprecated feature flags. The impact resulted in users receiving 404 error messages when attempting to access their training or log in to the Learner Experience. ‌ Multiple customers experienced an inability to access the platform, as initially reported on November 14, 2025, at 4:35 p.m. \(UTC\). The issue was identified quickly, mitigated, and resolved on November 14, 2025, at 5:07 p.m. \(UTC\). # WHAT HAPPENED The Learner Experience \(LX\) system relies on a modern frontend architecture that communicates with backend services via GraphQL to retrieve user attributes and configurations. This ensures that learners see the correct training content, risk scores, and interface options based on their organization's settings. ‌ On November 14, 2025, engineering teams executed a planned deployment to clean up the codebase by removing legacy code and feature flags. This change involved removing specific columns from the backend database schema that were deemed obsolete and removing them from all related frontend services. ‌ Immediately following this deployment, monitoring systems and customer support tickets indicated a spike in 404 errors for users attempting to load the Learner Experience. Investigations revealed that, although the backend columns had been successfully removed, the frontend application still retained a dependency on these fields. Specifically, the frontend was still attempting to query the now-deleted columns via GraphQL. This schema mismatch caused the API calls to fail entirely, resulting in the 404 errors presented to end users. # ROOT CAUSE ### **Primary Root Cause: Unsynchronized Deprecation** The backend deployment removed a database column that the frontend application was still actively querying. The deployment of the backend change preceded the removal of the frontend dependency, resulting in a breaking change to the GraphQL schema contract. ### **Secondary Root Cause: Post-Deployment Verification Latency** While automated smoke tests successfully identified the failure, they are configured to run immediately after deployments to production. Consequently, the invalid state was already live and impacting users by the time the test results confirmed the failure. # FINDINGS AND MITIGATIONS **Frontend Dependency on Deleted Schema** * **Finding:** The frontend application requests a specific field, which was mapped to the related column that was removed in the backend deployment. * **Mitigation:** Identified the failing GraphQL query and deployed a hotfix to remove the mapped column reference from the Learner Experience service. This hotfix restored login access for all users. **Observability and Alerting** * **Finding:** The issue was detected rapidly via "Sev0" alerts and support tickets. The post-deployment smoke tests also failed, confirming the issue, though this occurred concurrently with customer reports. * **Mitigation:** Teams pinpointed the exact deployment and merge request responsible, allowing for a rapid rollback/fix decision. # TECHNICAL DETAILS The Learner Experience frontend utilizes GraphQL to hydrate user sessions with configuration data. The specific failure occurred when the application requested the `$attributes` object, which included the field for `reportsConfiguration.riskScoreV2Enabled`. ‌ When the backend deployment \(aimed at removing these deprecated flags\) completed, the database schema no longer supported these fields. However, the frontend query had not yet been updated to stop requesting them. When the API received a request for fields that no longer existed in the schema definition, it returned a validation error. The application, unable to handle this critical failure during the initial load sequence, defaulted to a 404 error page. # PREVENTIVE MEASURES 1. **Phased Schema Deprecation:** Future removal of database columns will follow a stricter "Deprecate -> Detach -> Remove" lifecycle. The frontend dependency must be removed and verified in production before the backend column is physically dropped. 2. **Dependency Audits:** We will implement stricter code search practices during cleanup initiatives to ensure all references \(frontend, backend, and reporting\) to a deprecated field are identified prior to removal. # CUSTOMER IMPACT AND RECOVERY On November 14, 2025, for a short period, customers attempting to access the Learner Experience or complete training received 404 errors, making it temporarily unavailable. Following the deployment of the hotfix, user access to the Learner Experience was restored quickly. No data was lost during this incident, and full operational integrity was confirmed shortly after. # CONCLUSION This incident highlights the critical importance of synchronized management between frontend dependencies and backend schema changes. While the intention to reduce technical debt by removing legacy flags was correct, the execution sequence caused a temporary service disruption. As a result of this RCA: * The problematic dependencies have been fully removed. * We have successfully cleaned up the legacy code, preventing future confusion or issues related to these fields. We are committed to ensuring the reliability of the Learner Experience and appreciate your patience as we resolved this matter.

Read the full incident report →

Started Nov 07, 2025, 09:57 PM UTC· resolved Nov 08, 2025, 02:13 PM UTC

1. investigating Nov 07, 2025, 09:57 PM UTC

   We are currently investigating issues when authenticating into Prevent, Protect and Defend.

2. investigating Nov 07, 2025, 10:03 PM UTC

   We are continuing to investigate this issue.

3. monitoring Nov 07, 2025, 11:08 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 08, 2025, 02:13 PM UTC

   This incident has been resolved.

5. postmortem Dec 10, 2025, 05:16 PM UTC

   On November 7, 2025, we identified an issue that prevented some customers from accessing the Defend Admin Portal using SAML-based authentication. This issue affected customers in the US region between approximately 3:32 p.m. UTC and 11:06 p.m. UTC on November 7, 2025. Customers using OAuth authentication were not affected. This issue was caused by an inadvertent configuration change to a critical system policy that removed a necessary value. When customers attempted to log in via SAML, the missing value caused authentication to fail. To resolve this issue, we rolled back the policy to its previous configuration. Customers were able to log in successfully at 11:02 p.m. UTC, and the incident was fully resolved at 11:06 p.m. UTC. To prevent similar issues in the future, we've implemented enhanced change control processes for critical system policies, including mandatory peer review requirements and additional confirmation steps before modifications can be made. We've also established a version control system to ensure quick recovery capability for all policies. No data loss occurred as a result of this issue.

Read the full incident report →

Started Nov 03, 2025, 03:14 PM UTC· resolved Nov 04, 2025, 12:34 PM UTC

1. investigating Nov 03, 2025, 03:14 PM UTC

   We have received reports of errors when using the Deployment Center for both Defend and Prevent . We are investigating this issue and will update this page when we have more information.

2. identified Nov 03, 2025, 06:05 PM UTC

   We've identified the cause of the issue and are working on implementing a fix. We'll continue to post on our status page with any new information or updates.

3. monitoring Nov 03, 2025, 06:50 PM UTC

   We’ve implemented a fix for the Deployment Center and we’re monitoring the results to make sure no further issues occur.

4. monitoring Nov 03, 2025, 06:52 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Nov 04, 2025, 12:34 PM UTC

   This incident has been resolved.

6. postmortem Dec 08, 2025, 03:28 PM UTC

   On Monday, November 3, from approximately 2:54 p.m. \(UTC\) until Tuesday, November 4, 2025, at 10:04 a.m. \(UTC\), some users were unable to deploy any instances of Defend or Prevent via the deployment center. This incident was caused by an intermittent failure when communicating with Microsoft systems while attempting to provision new groups within customer tenants as part of the product deployment process. To resolve this issue, we changed the group creation process to use Graph API rather than Exchange PowerShell, which allowed the groups to be successfully created within customer tenants. All affected deployments were subsequently retried by customers and completed successfully. To prevent this type of issue in the future, we have implemented additional monitoring for the deployment process. No data loss occurred as a result of this issue.

Read the full incident report →

Started Oct 29, 2025, 04:22 PM UTC· resolved Oct 30, 2025, 01:42 PM UTC

1. investigating Oct 29, 2025, 04:22 PM UTC

   We are investigating elevated error rates and loading issues across multiple products. Initial investigations point to a widespread Azure incident. More info will be added after our engineering teams are able to assess the scope of the issues.

2. investigating Oct 29, 2025, 04:35 PM UTC

   We are continuing to investigate this issue.

3. investigating Oct 29, 2025, 04:37 PM UTC

   We are continuing to investigate this issue.

4. investigating Oct 29, 2025, 04:46 PM UTC

   We are continuing to investigate this issue.

5. investigating Oct 29, 2025, 05:06 PM UTC

   We are continuing to investigate this issue.

6. investigating Oct 29, 2025, 05:17 PM UTC

   We are continuing to investigate this issue.

7. investigating Oct 29, 2025, 05:18 PM UTC

   We are continuing to investigate this issue.

8. investigating Oct 29, 2025, 05:28 PM UTC

   We continue to investigate elevated error rates and loading issues across multiple products. This outage is due to a widespread Azure incident. More info will be added after our engineering teams are able to assess the scope of the issues. Further information can be found on Microsoft's status page: https://azure.status.microsoft/en-gb/status

9. identified Oct 29, 2025, 07:49 PM UTC

   Microsoft has performed a rollback to their "Last Known Good" configuration and we hope to see signs of recovery soon. Note: the upstream Azure outage is also impacting a Microsoft-hosted office.js library, resulting in loading issues with Hybrid/Ribbon PAB and the KnowBe4 Email Security Add-in. We'll continue to post on our status page with any new information or updates. Further information can be found on Microsoft's status page: https://azure.status.microsoft/en-gb/status

10. identified Oct 29, 2025, 09:49 PM UTC

    We are continuing to work on a fix for this issue.

11. monitoring Oct 30, 2025, 12:11 AM UTC

    While our systems are back to operational, we are continuing to monitor the health of the overall platform and all its subcomponents to ensure stability. Further information can be found on Microsoft's status page: https://azure.status.microsoft/en-gb/status

12. resolved Oct 30, 2025, 01:42 PM UTC

    This incident has been resolved.

13. postmortem Nov 04, 2025, 04:48 PM UTC

    External Technical Root Cause Analysis: Microsoft Azure Front Door Outage Impact on KnowBe4 Products October 29, 2025 # Summary This report details the findings and mitigations related to a global Microsoft Azure Front Door \(AFD\) outage that occurred between 15:45 UTC on October 29 and 00:05 UTC on October 30, 2025. The incident disrupted multiple Microsoft services — including Azure Active Directory, Azure Portal, Microsoft Entra ID, and Office 365 add-in infrastructure — which, in-turn, directly affected several KnowBe4 products that depend on these services for routing, authentication, and Office add-in initialization. While user interfaces and administrative consoles experienced intermittent unavailability, all backend mail-processing and delivery systems within KnowBe4 remained operational and continued to function as designed. The outage originated entirely within Microsoft’s Azure Front Door global network, following an inadvertent tenant-configuration deployment. The faulty configuration propagated inconsistently across AFD nodes worldwide, causing widespread routing errors and service timeouts until Microsoft rolled back to a last-known-good state. # Glossary of Technical Terms **Azure Front Door \(AFD\):** Microsoft’s global, scalable content delivery network \(CDN\) and web application firewall \(WAF\) service used to route internet traffic securely and efficiently to Azure-hosted applications. **Azure Front Door Configuration:** A set of routing and security rules that define how traffic flows through Azure Front Door to customer applications. A configuration error or invalid deployment can cause global routing disruptions. **Azure Active Directory / Microsoft Entra ID:** Microsoft’s identity and access management service that provides user authentication, single sign-on, and authorization across Azure and Microsoft 365 applications. **Office Web Add-in:** A lightweight web application that extends Microsoft Outlook, Excel, Word, and other Office products. KnowBe4’s add-ins use this framework to provide in-app phishing-reporting and email-classification functionality. [**appsforoffice.microsoft.com**](http://appsforoffice.microsoft.com)**:** A Microsoft-controlled endpoint that serves the Office.js library, which is required to initialize all Office web add-ins. If unavailable, no add-in code can execute. **Office.js:** The client-side JavaScript framework that provides APIs for Office web add-ins. It must load successfully from Microsoft’s servers before an add-in can function. **Last Known Good Configuration:** The most recent stable configuration version used by Microsoft to restore service after an invalid deployment or system failure. **Web Application Firewall \(WAF\):** A network security layer that filters and monitors HTTP\(S\) traffic between a web application and the Internet, protecting against common web exploits and attacks. **Mailflow:** The process by which emails are received, analyzed, classified, and delivered within KnowBe4’s mail-security products. This component was unaffected during the Azure outage. **Magic Link:** A secure, single-use authentication link that allows KnowBe4 administrators or new customers to access a portal or begin deployment without entering credentials. **Telemetry:** Automated data collection used for monitoring system performance and availability, enabling detection of latency, errors, or external dependency issues. **Fail-Open:** A system design approach in which, during an outage or dependency failure, limited functionality remains available instead of the service fully blocking operations. # WHAT HAPPENED ## Azure Infrastructure Outage At 15:45 UTC on October 29, 2025, Microsoft Azure Front Door \(AFD\) experienced a global service disruption caused by an inadvertent tenant configuration deployment that introduced an invalid state across the AFD network. The invalid configuration caused a significant number of nodes to fail to load properly, resulting in widespread latencies, timeouts, and connection errors for both Microsoft services and customer applications. As unhealthy nodes were dropped from service, traffic was automatically rerouted to the remaining healthy nodes, creating further imbalances and intermittent regional outages. Microsoft responded by blocking all configuration changes, deploying a “last known good” configuration, and manually recovering the affected nodes. Full mitigation was confirmed by Microsoft at 00:05 UTC on October 30, 2025, following the gradual rebalancing of global traffic. During the outage window, dependent services such as Microsoft Entra ID, Office 365 add-in frameworks \([appsforoffice.microsoft.com](http://appsforoffice.microsoft.com)\), and the Azure Portal were also impacted. These dependencies are critical to KnowBe4’s authentication, routing, and Office-based integrations, and their unavailability directly contributed to product-level interruptions. ## Platform-Level Effects ### Prevent & Protect * Users of Microsoft Outlook add-ins for Prevent and Protect experienced hanging send processes and error messages such as “Outlook can’t send this message because there’s a problem with the add-in.” * Because the Microsoft-hosted Office.js library could not initialize, the task pane failed to load entirely, leaving users unable to send or classify emails. * Gateway Moderation features and backend mailflow continued to function normally. ### Defend * Administrators were unable to log in to the Defend console, which relies on the KnowBe4 Security Center \(KSC\) for authentication. * KSC, in turn, uses Azure Front Door and Microsoft Entra ID for routing and identity validation. * Although the Defend console was unavailable, core email processing, analysis, and delivery pipelines remained fully operational. ### Phish Alert Button \(PAB – Hybrid & MSR\) * Users attempting to launch the Phish Alert Button \(PAB\) encountered indefinite loading and initialization failures. * Because the add-in could not reach [appsforoffice.microsoft.com](http://appsforoffice.microsoft.com), the Office.js script never executed, preventing the add-in from rendering or reporting emails. * PAB for Gmail users were unaffected. ### KnowBe4 Security Center \(KSC\) * The KSC administrative dashboard became temporarily inaccessible due to its reliance on Azure Front Door and Microsoft Entra ID for authentication and session management. * All backend data collection and product reporting pipelines remained intact. ### Webforms * Customer-facing webforms hosted through Azure were inaccessible during the incident. * Because Azure Front Door manages TLS certificate termination and inbound routing, form submissions and UI access were blocked until Microsoft restored AFD. ### KnowBe4 Deployment Center * Customers were unable to receive magic links or access onboarding interfaces during the outage. * In-progress deployments may have failed to complete due to API call disruptions with Microsoft Graph and Exchange Online. * Once service was restored, administrators could retry affected deployments with no lasting customer impact. ### Workspace * No confirmed customer impact. Active sessions remained stable, and no user reports indicated degradation of service during the outage window. # ROOT CAUSE ANALYSIS The Azure Front Door outage was caused by an inadvertent tenant configuration change within the AFD internals that introduced an invalid state to AFD nodes globally. A software defect in Microsoft’s deployment validation process allowed the faulty configuration to bypass safety checks and propagate to production, causing nodes to fail to load properly and drop from the network. As nodes failed, traffic was rerouted to remaining healthy regions, overloading them and amplifying global latency and timeout rates. Microsoft blocked further configuration changes, rolled back to the previous known-good state, and manually rebalanced traffic to restore stability. Because some of KnowBe4’s services depend on Azure Front Door for routing and Microsoft’s Office 365 endpoints \(e.g., [appsforoffice.microsoft.com](http://appsforoffice.microsoft.com)\) for add-in initialization, no failover mechanism could circumvent this Microsoft-controlled dependency during the event. ## Scope of Impact Impact systems globally using these foundational services and followed Microsoft’s global impact window \(15:45 UTC, Oct 29 – 00:05 UTC, Oct 30\): Prevent & Protect Outlook web add-ins failed to load; email send operations hung with error messages. Gateway Moderation unaffected. Defend Admin console login unavailable due to Front Door authentication dependency; email processing unaffected. PAB – Hybrid & MSR Add-in failed to initialize via appsforoffice.microsoft.com; Gmail extension unaffected. KSC Reporting UI and login unavailable; no mailflow impact. Webforms External forms temporarily inaccessible as certificates and routing were managed via Front Door. KnowBe4 Deployment Center Unable to request new magic links or access onboarding UI; some in-progress deployments failed but were recoverable after service restoration. Workspace No confirmed customer impact; active sessions remained stable. No customer data loss occurred\*\*,\*\* and normal operation resumed once Microsoft completed its rollback. ## Detailed Timeline \(UTC\) Oct 29 15:45 Microsoft Azure Front Door Impact Begins Global service disruption begins after an inadvertent tenant configuration change within Azure Front Door \(AFD\). Customers and Microsoft services begin to experience latency, timeouts, and connectivity errors. 16:04 Microsoft Investigation Initiated Azure monitoring systems trigger alerts. Microsoft engineers begin reviewing recent AFD configuration changes. 16:15 Root-Cause Isolation Begins Microsoft identifies the likely source as a tenant configuration deployment that entered an invalid state across global nodes. 16:18 First Public Communication Posted by Microsoft Initial Microsoft status update published to Azure Status page. Internal KnowBe4 monitoring detects elevated timeouts across Office web add-ins and administrative consoles. 16:20 Targeted Notifications Issued Microsoft sends targeted impact notifications through Azure Service Health. KnowBe4 teams begin correlation of customer reports with AFD dependency failures. 16:22 KnowBe4 Publishes StatusPage Alerts Acknowledging Outage “We are investigating elevated error rates and loading issues across multiple products. Initial investigations point to a widespread Azure incident. More info will be added after our engineering teams are able to assess the scope of the issues.” \(Status Page\) 17:26 Azure Portal Fails Away from Front Door Microsoft routes Azure Portal traffic off AFD as part of mitigation. KnowBe4 observes continuing errors across products using AFD endpoints. 17:30 Configuration Changes Blocked Globally Microsoft freezes all customer and internal configuration updates to stop further propagation of the faulty state. 17:40 ‘Last Known Good’ Configuration Deployed Microsoft initiates rollout of the most recent validated configuration across AFD infrastructure. KnowBe4 confirms stabilization in limited regions. 18:30 Global Configuration Push Begins Fixed configuration distributed worldwide. Gradual traffic recovery observed. KnowBe4 services begin to auto-recover region by region. 18:45 Manual Node Recovery and Rebalancing Microsoft engineers start manually recovering affected AFD nodes and gradually restore routing to healthy nodes. 19:00 KnowBe4 Internal Status Update Engineering publishes initial internal assessment: backend mailflow unaffected, user-facing add-ins and consoles unavailable. 20:30 Partial Recovery Confirmed Microsoft reports substantial improvement in latency and error rates. KnowBe4 verifies restored access for portions of US and EU customer traffic. 23:15 Downstream Dependencies Stabilize Microsoft confirms mitigation for PowerApps and related services. KnowBe4 validates near-normal operation across most regions. Oct 30 00:05 Global Mitigation Complete Microsoft confirms full restoration of Azure Front Door and Office 365 add-in endpoints. KnowBe4 services fully recovered with no remaining customer impact. # FINDINGS AND MITIGATIONS ### 1. Azure Front Door Dependency **Finding:** All impacted KnowBe4 applications use Azure Front Door for secure global routing, TLS termination, and authentication relay.**Mitigations:** * Continued engagement with Microsoft for further clarity on resilience plans and post-mortem recommendations. ### 2. Microsoft Office Add-In Initialization **Finding:** Office web add-ins are hard-coded to initialize through [appsforoffice.microsoft.com](http://appsforoffice.microsoft.com) before executing customer code.**Mitigations:** * Request Microsoft to consider manifest-level fail-open or offline initialization mechanisms. * Internal research into safe caching of Office.js to improve resilience testing \(acknowledging Microsoft does not currently support self-hosting\). ### 3. Monitoring and Communications **Finding:** The outage was initiated within Microsoft’s control plane and communicated via Azure Status and Service Health dashboards.**Mitigations:** * Enhanced internal telemetry to detect third-party latency patterns earlier. * Streamlined incident communication templates to accelerate customer notifications during vendor outages. # CUSTOMER IMPACT Between 15:45 UTC on October 29 and 00:05 UTC on October 30, 2025, customers experienced periodic errors, timeouts, and login failures across Microsoft-integrated KnowBe4 interfaces. Administrative consoles, add-ins, and forms were unavailable for segments of this window, depending on regional propagation of the faulty AFD configuration. Backend systems—including Prevent, Protect, Defend, and PhishER pipelines—continued to deliver mail security and training events without interruption. Following Microsoft’s rollback and global node rebalancing, service availability gradually returned to normal with no customer action required. # PREVENTIVE MEASURES ## Improved Internal Monitoring and Observability To accelerate detection and correlation of third-party dependency failures, new synthetic monitoring tests have been added to continuously validate access to [appsforoffice.microsoft.com](http://appsforoffice.microsoft.com), Azure Front Door endpoints, and related Microsoft authentication paths. ## Add-in and Client Resilience Testing Although Office web add-ins cannot currently bypass Microsoft’s initialization endpoint, KnowBe4 is: * Conducting research into safe caching of Office.js libraries for limited resilience testing. * Exploring options to detect and message users when Microsoft dependencies are unavailable, improving transparency during third-party outages. ## Customer Communication Enhancements KnowBe4 has updated its incident-response and communication templates to provide faster, clearer updates when external infrastructure failures occur. * This includes standardized language to distinguish between core service availability \(mailflow and backend processing\) and third-party integration impacts \(e.g., Office add-ins, Cloud Provider\(s\) or consoles\). ## **CONCLUSION** This incident was caused by a Microsoft-initiated tenant configuration error within Azure Front Door, which propagated globally and temporarily disrupted routing and authentication services for Microsoft and dependent applications. KnowBe4’s platform maintained operational integrity and security throughout the event; however, user-facing Microsoft-dependent interfaces were interrupted until Azure Front Door was fully restored. KnowBe4 continues to work with Microsoft to understand root-cause remediation and to evaluate resilience enhancements for critical dependencies. We remain committed to transparency, reliability, and continuous improvement of our service availability for all customers.

Read the full incident report →

Started Oct 20, 2025, 09:47 AM UTC· resolved Oct 21, 2025, 01:57 PM UTC

1. monitoring Oct 20, 2025, 09:47 AM UTC

   We experienced degraded performance across parts of the platform earlier today due to an AWS service incident in the US-EAST-1 region. Service levels are improving, and our teams continue to monitor the situation closely.

2. monitoring Oct 20, 2025, 10:15 AM UTC

   We are continuing to monitor for any further issues.

3. monitoring Oct 20, 2025, 10:19 AM UTC

   We are continuing to monitor for any further issues.

4. monitoring Oct 20, 2025, 12:00 PM UTC

   Phish Alert, Secure Workspace, and Defend Mail Flow are operational again. We are continuing to monitor PhishER and KSAT.

5. monitoring Oct 20, 2025, 12:06 PM UTC

   We are continuing to monitor for any further issues.

6. monitoring Oct 20, 2025, 08:06 PM UTC

   We continue to experience intermittent degraded performance across parts of the platform due to an AWS service incident in the US-EAST-1 region. Service levels continue to improve, and our teams continue to monitor the situation closely.

7. resolved Oct 21, 2025, 01:57 PM UTC

   The performance issues impacting our platform have been addressed and this incident has been resolved. Our team will be able to provide a root cause analysis once we have received full details on the service event incident from AWS.

8. postmortem Oct 27, 2025, 03:32 PM UTC

   # External Technical Root Cause Analysis: Service Disruptions October 20, 2025 # Summary On Monday, October 20, 2025, multiple KnowBe4 products, including Defend, KSAT, and PhishER, experienced major service degradation following a regional AWS outage in US-EAST-1. According to[ AWS’s official post-event summary](https://aws.amazon.com/message/101925/), the root cause was a race condition in the control-plane DNS infrastructure for Amazon DynamoDB, which prevented proper DNS resolution for DynamoDB endpoints. This failure caused a cascade of dependency failures across several AWS services that KnowBe4 leverages for delivery of our products, including ECS, Lambda, SQS, and Networking, all of which depend on DynamoDB for state management and metadata coordination. The disruption led to widespread queue accumulation, container launch failures, and elevated latency across KnowBe4’s systems. Basic platform functionality \(web navigation, login, and API access\) was restored for most services by 18:15 \(UTC\), and full service restoration was completed by 22:00 \(UTC\), with no data loss reported. # Glossary of Technical Terms _The following terms are provided for clarity and reference._ **ECS \(Elastic Container Service\)** – An AWS service for hosting containers, the lightweight “servers” that KnowBe4 uses to process website traffic, background jobs, and campaign tasks. **Lambda** – AWS’s serverless compute platform that runs small, event-driven functions without needing a full-blown server. KnowBe4 uses this service for short-lived jobs like message classification and workflow automation. **DynamoDB** – AWS’s fully managed NoSQL database. Many AWS services \(and some KnowBe4 components\) depend on it for storing state and metadata used to coordinate system activity. **SQS \(Simple Queue Service\)** – A message-queuing system that temporarily stores jobs or events between services. KnowBe4 uses it to handle spikes in workload safely and ensure reliable event delivery. **Control Plane** – The set of management systems that create, update, and coordinate resources \(e.g., starting containers or functions\). If the control plane fails, new resources can’t launch even if old ones keep running. **DNS \(Domain Name System\)** – The internet’s “address book”, translating names like `training.knowbe4.com` into network addresses. A DNS failure can prevent services from finding each other inside AWS. **Retry Storm** – A sudden surge of repeated attempts to reach a failed service. During outages, these retries can flood systems and make recovery slower. **Queue Backlog** – A term used to describe when many jobs are piled up waiting for processing. This happened in Defend, KSAT, and PhishER during the outage. # WHAT HAPPENED ## AWS Infrastructure Outage At 06:49 \(UTC\) on October 20, 2025, AWS observed elevated error rates across multiple services within the US-EAST-1 region. By 07:26 \(UTC\), AWS identified the issue as DNS resolution failures affecting Amazon DynamoDB endpoints, caused by a race condition in the DynamoDB control-plane DNS infrastructure. The fault disrupted DynamoDB’s ability to resolve internal service addresses, leading to widespread dependency failures across AWS services that rely on it for metadata and state management. AWS mitigated the underlying DNS race condition by 09:24 \(UTC\), restoring core DynamoDB functionality. However, extensive retry traffic and queued requests from dependent services \(including ECS, Lambda, and SQS\) continued to overload control-plane capacity for several hours. AWS applied rate-limiting and phased service restarts to stabilize the region and prevent further saturation. Because many of KnowBe4’s workloads depend on ECS and Lambda orchestration, the outage manifested as container and task launch failures across Defend, KSAT, and PhishER. Existing workloads largely remained functional but were unable to scale or replace failed tasks. As AWS restored regional control-plane stability, KnowBe4 systems began recovering, with basic functionality \(web navigation, login, API access\) available again by 18:15 \(UTC\) and full service restoration completed by 22:00 \(UTC\), with no data loss reported. ## Platform-Level Effects ### Defend #### US Region * At 07:30 \(UTC\), Defend received alerts indicating service degradation. DNS health checks failed open due to unavailable AWS metrics affecting both US-EAST-1 and US-WEST-2. Although automatic failover executed successfully, the absence of health-check metrics caused traffic to remain evenly split between regions rather than fully shifting west. * By 10:00 \(UTC\), AWS metrics had recovered, allowing new traffic to route exclusively to US-WEST-2. A small number of messages that had already passed the load balancer in US-EAST-1 were temporarily queued and unable to complete processing while the regional AWS issues persisted. * All queued messages were successfully processed by 11:56 \(UTC\), and message throughput returned to normal. A limited subset of customers experienced delayed message processing, while the majority were unaffected thanks to the partial success of automatic failover. #### UK Region * At 07:30 \(UTC\), AWS metric outages caused DNS health checks in the UK region to fail open. Engineering teams identified a legacy DNS record that was incorrectly evaluated as the healthiest endpoint, returning inactive IP addresses to clients. These addresses no longer pointed to live infrastructure, causing SMTP connection failures across all mail traffic. * Once AWS restored console access around 09:30 \(UTC\), engineers deployed a configuration change to remove the legacy record. Microsoft’s mail infrastructure automatically retried previously failed deliveries between 09:30 and 10:30 \(UTC\), and traffic volumes had normalized by 10:30. * All SMTP customers in the UK region experienced temporary service disruption, while customers using Microsoft Graph API for mail delivery were unaffected. ### KSAT * KSAT web services, campaign dispatchers, and enrollment workers failed to provision new tasks from 06:50 through 18:00 \(UTC\), resulting in delayed campaign launches, notification emails, and user enrollments. * KSAT APIs and the web console experienced intermittent timeouts and latency until the DynamoDB and ECS control-plane operations recovered. * Recovery began around 18:00 \(UTC\), with basic product functionality restored by 18:15 \(UTC\) and backlog queues clearing completely around 21:45 \(UTC\). * All training and phishing campaigns resumed automatically, with no queued data lost. ### PhishER * PhishER’s email-categorization and YARA rule-evaluation workers, which rely on Lambda and SQS orchestration, began failing at 06:50 \(UTC\), producing ingestion and processing backlogs. * Message classification latency increased significantly throughout the morning as Lambda provisioning errors persisted. * AWS service recovery between 18:00 and 19:00 \(UTC\) allowed task provisioning to resume and queues to drain. * Full message flow was restored by 20:55 \(UTC\), with zero data loss. ## Secondary Issue: Email Delivery * During the AWS service disruption, KnowBe4’s phishing campaign mail senders were affected by the same control-plane limitations that prevented new ECS task launches. * To restore outbound campaign capacity, the mail service was temporarily reconfigured to operate using an alternate networking path that bypassed normal IP-restriction controls. * On October 20, from 18:06 UTC until October 21 at 16:33 UTC, approximately 80,000 phishing simulation emails were delivered from a KnowBe4-owned backup IP address \(34.199.159.209\) that is not published on the official allowlist for phishing mail delivery. This IP does appear in other internal documentation \(such as the Syslog whitelist\) but is not intended for use with phishing campaigns. * As a result, some recipients may have experienced false-positive click activity or mail filtering inconsistencies. The issue was identified and resolved the day after the primary outage. * Customers are advised not to whitelist the backup IP and to contact support if discrepancies appeared in campaign metrics. # ROOT CAUSE ANALYSIS The service disruptions on October 20, 2025 originated from a race condition in the control-plane DNS infrastructure for Amazon DynamoDB in the US-EAST-1 region, preventing proper DNS resolution for DynamoDB endpoints and disrupting multiple dependent AWS services — most notably ECS, Lambda, and SQS — that rely on DynamoDB for metadata and state management. As a result, new container and function launches failed across US-EAST-1, while existing workloads generally remained operational but could not scale or replace failed tasks. AWS mitigated the core DNS race condition by 09:24 \(UTC\), but cascading retry storms and queued requests from dependent services prolonged instability through the morning. AWS applied rate-limiting and phased restarts to gradually restore control-plane capacity. For KnowBe4, this manifested differently across products and regions: * In Defend \(US region\), DNS health checks failed open due to unavailable AWS service metrics, causing the global traffic manager to split requests roughly 50/50 between US-EAST-1 and US-WEST-2 instead of fully failing over. Although automatic failover executed successfully, missing AWS metrics prevented it from completing cleanly. Once AWS metrics returned \(approximately 10:00 \(UTC\)\), traffic shifted fully to US-WEST-2, and remaining messages queued in US-EAST-1 processed by 11:56 \(UTC\). * In Defend \(UK region\), AWS console and metric unavailability led to a legacy DNS record being erroneously evaluated as healthy. This caused mail traffic to resolve to inactive IPs, resulting in SMTP connection failures from 07:30 - 10:30 \(UTC\). Once AWS restored console access, engineers removed the legacy record, and Microsoft retried failed deliveries automatically. * In KSAT and PhishER, failures in ECS, Lambda, and SQS prevented new task provisioning from 06:50 \(UTC\) onward, leading to significant backlog accumulation. Basic functionality across most products returned by 16:15 \(UTC\), and all queued workloads cleared successfully by 22:00 \(UTC\), with no data loss. The underlying cause of the incident was entirely within AWS’s managed infrastructure. AWS has since implemented changes to the DynamoDB DNS subsystem to remove the race condition and improve isolation of control-plane workloads. On KnowBe4’s side, the event exposed dependencies between AWS regional metrics, DNS health checks, and traffic routing logic that can complicate failover when control-plane telemetry becomes unavailable. Internal mitigations have been applied to improve observability, region-aware failover logic, and operational readiness during third-party service disruptions. ## Detailed Timeline \(UTC\) ‌ | **Time \(UTC\)** | **Event** | | --- | --- | | **06:49** | AWS identifies elevated error rates in US-EAST-1; DynamoDB control-plane DNS race condition begins, causing service failures in Defend, KSAT, PhishER. | | **07:30** | Defend US and UK regions detect DNS health-check anomalies; automatic failover partially executes. AWS metrics unavailable. | | **08:30 - 09:30** | Engineers identify and remove legacy DNS record after AWS console access is restored; Microsoft retries failed SMTP deliveries. | | **09:24** | AWS mitigates the underlying DynamoDB DNS issue but dependent services \(ECS, Lambda, SQS, ECR, ENI\) remain impaired due to retry traffic and backlogs. | | **10:00** | Defend US routing fully shifts to US-WEST-2; residual queued messages remain in US-EAST-1 awaiting processing. | | **11:56** | All queued Defend US messages processed; Defend UK and US regions fully stable. | | **18:00 - 19:00** | AWS control-plane stability largely restored; ECS/Lambda task launches resume across KSAT and PhishER; backlogged jobs begin processing normally. | | **20:55** | PhishER message-flow normalization completed; PhishER appears to be healthy. | | **21:45** | KSAT queues are fully processed; all campaigns and enrollments complete successfully; KSAT appears to be healthy. | | **22:00** | All KnowBe4 services confirmed operational across regions; full recovery achieved with no data loss. | ‌ # FINDINGS AND MITIGATIONS ## 1. AWS Control-Plane Instability A race condition in the DynamoDB control-plane DNS infrastructure in US-EAST-1 caused widespread dependency failures across AWS services \(ECS, Lambda, SQS, ECR, ENI\). The failure prevented new resource launches, degraded AWS metrics visibility, and disrupted hosted applications globally. Mitigations \(AWS\): * AWS resolved the DNS race condition by 09:24 \(UTC\) and implemented architectural changes to eliminate single-threaded DNS dependencies in DynamoDB’s control plane. * Introduced improved isolation boundaries between control-plane workloads and enhanced retry and back-pressure logic to prevent saturation from cascading retry traffic. * Committed to improved monitoring and internal alerting for cross-service dependency failures. Mitigations \(KnowBe4\): * Adjusted internal service scaling and SQS polling intervals to avoid excessive retries during third-party degradation. * Updated dependency tracking to recognize AWS service health events earlier in our observability stack. * Enhanced escalation procedures for AWS-originating regional incidents. ## 2. Defend Regional Failover and DNS Dependencies During the outage, AWS metric unavailability caused DNS health checks for Defend to fail open. This resulted in incomplete automatic failover between US-EAST-1 and US-WEST-2, and in the UK region, a legacy DNS record was incorrectly evaluated as healthy, directing SMTP traffic to inactive endpoints. Mitigations: * Updated failover logic to include fallback routing when AWS health metrics are unavailable. * Implemented automated alerts for DNS records that reference inactive infrastructure. * Strengthened the DNS validation process to prevent reintroduction of deprecated records. * Verified that Graph-based mail routing remains isolated from DNS-based health metrics. ## 3. Queue Backlog and Job Prioritization Outage conditions caused millions of delayed jobs across KSAT and PhishER queues. ECS and Lambda provisioning failures prevented new workers from being launched, and retry behavior risked additional load on recovering infrastructure. Mitigations: * Implemented priority-aware scaling logic to allow controlled queue draining once AWS service capacity resumes. * Introduced additional visibility metrics for queue length, processing throughput, and per-region lag. * Added safeguards to prevent retry amplification during partial recoveries. ## 4. Temporary Email Delivery Workaround To restore phishing-campaign delivery during the outage, a temporary alternate network configuration was used, resulting in approximately 80 000 messages being sent from an unlisted IP address \(34.199.159.209\). This created potential for false-positive clicks and mail-filtering inconsistencies. Mitigations: * Issue identified and configuration reverted by October 21, 2025, at 16:33 \(UTC\). * Customers notified not to whitelist the backup IP; monitoring added to flag non-approved sender routes. * Engineering has initiated a project to establish pre-approved, resilient failover paths for outbound mail that maintain customer-visible IP consistency. # CUSTOMER IMPACT ## Defend - US Region * A subset of customers experienced delayed message processing between 07:30 and 11:56 \(UTC\) due to queued traffic in US-EAST-1. * Automatic failover to US-WEST-2 executed successfully but operated at a partial 50/50 traffic split until AWS metrics recovered at 10:00 \(UTC\). * The majority of customers experienced no outage, and all queued messages were successfully processed with no loss of telemetry or data. ## Defend - UK Region * From 07:30 to 10:30 \(UTC\), customers using SMTP-based mail delivery experienced a complete service interruption caused by DNS mis-routing to inactive infrastructure. * Customers using Microsoft Graph API for email delivery were not affected. * Microsoft’s automated retry mechanisms successfully delivered all previously failed messages once the DNS configuration was corrected. ## KSAT * Customers experienced delayed phishing campaign launches, training enrollments, and notification deliveries from 06:50 through 21:45 \(UTC\). * Platform login, dashboard, and API access were intermittently degraded until 16:15 \(UTC\), when basic functionality was restored. * All queued campaigns resumed automatically; no user data or campaign content was lost. ## PhishER * Customers observed slower email classification, rule evaluation, and workflow execution from 06:50 through 20:55 \(UTC\), corresponding to Lambda provisioning failures and SQS backlog buildup. * All queued messages were processed successfully, with zero data loss or permanent rule failures. ## Email Delivery \(Secondary Issue\) * On October 20, from 18:06 \(UTC\) until October 21 at 16:33 \(UTC\), approximately 80 000 phishing simulation emails were sent from a KnowBe4-owned backup IP address \(34.199.159.209\) not listed on the standard allowlist. * This temporary measure allowed campaigns to continue sending but introduced a risk of false-positive clicks and mail-filtering inconsistencies for some customers. * The configuration was reverted on October 21, and impacted customers were notified and advised not to whitelist the backup IP. # PREVENTIVE MEASURES * **Region-Aware Failover Improvements**Updated Defend’s failover logic to better handle conditions where AWS metrics or health checks are unavailable. DNS failover can now evaluate multiple signals, ensuring full traffic diversion when metric sources fail open. * **Cross-Region Deployment Validation**Added automated verification of DNS and health-check configurations in all active Defend regions to prevent legacy or inactive endpoints from being marked as healthy. * **Priority-Aware Scaling and Queue Draining**Implemented job-aware scaling logic for ECS and Lambda workers to allow controlled queue recovery after service interruptions without overloading dependent systems. * **Retry Behavior Controls**Introduced stronger client-side back-off and exponential retry behavior in KnowBe4 systems to prevent retry amplification during upstream cloud instability. * **Mail Infrastructure Hardening**We will be implementing a project to expand mail delivery routes to ensure that we can maintain consistent IP reputation and eliminate the need for manual overrides during future regional events. ## **CONCLUSION** The service disruptions on October 20, 2025 demonstrated how a single upstream control-plane failure within AWS can cascade across dependent systems and regions. The race condition in the DynamoDB DNS infrastructure in US-EAST-1 triggered widespread instability in services fundamental to KnowBe4’s hosting architecture — particularly ECS, Lambda, and SQS — which in turn affected Defend, KSAT, and PhishER. Despite the scale of the AWS outage, KnowBe4’s platforms remained data-intact, and core functionality recovered within hours of AWS mitigation. Automatic failover mechanisms limited customer impact in the US region, while targeted engineering intervention restored full operation in the UK region once AWS console access returned. Queue-based systems such as KSAT and PhishER processed all backlogged jobs successfully, and no data loss occurred. The incident validated the recovery processes of our systems following severe third-party degradation while also revealing opportunities to improve scaling control and retry behavior during upstream cloud instability. The preventive measures coming out of this incident — enhanced failover logic, automated DNS validation, job-aware queue scaling, and mail infrastructure hardening — directly address those gaps and strengthen our ability to withstand future regional cloud events. KnowBe4 continues to monitor AWS’s remediation efforts related to this incident and will integrate any further best practices as they are published. Through these combined improvements, we are better positioned to maintain continuity, minimize customer impact, and ensure operational transparency during large-scale cloud disruptions.

Read the full incident report →

Started Oct 17, 2025, 06:07 PM UTC· resolved Oct 21, 2025, 09:12 PM UTC

1. identified Oct 17, 2025, 06:07 PM UTC

   We've identified the cause of training policies not launching and are working on implementing a fix. We'll continue to post on our status page with any new information or updates.

2. monitoring Oct 17, 2025, 06:21 PM UTC

   We’ve implemented a fix for training policies and we’re monitoring the results to make sure no further issues occur.

3. resolved Oct 21, 2025, 09:12 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 14, 2025, 01:41 PM UTC· resolved Oct 15, 2025, 01:43 PM UTC

1. investigating Oct 14, 2025, 01:41 PM UTC

   Customers may be unable to log in to their KSAT, SecurityCoach, or PhishER consoles. Additionally, this is causing initialization issues with the Phish Alert Button. We are investigating this issue and will update this page when we have more information.

2. identified Oct 14, 2025, 03:22 PM UTC

   We've identified the cause of the login and are working on implementing a fix for the US instance. We've applied a fix for the EU instance and are monitoring the results to make sure no further issues occur. We'll continue to post on our status page with any new information or updates.

3. identified Oct 14, 2025, 06:25 PM UTC

   We continue to work to resolve this issue. We have a few mitigating items in place and the console is loading successfully. We are beginning to process a backlog of campaigns

4. monitoring Oct 14, 2025, 07:10 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Oct 15, 2025, 01:43 PM UTC

   This incident has been resolved.

6. postmortem Oct 27, 2025, 02:18 PM UTC

   # External Technical Root Cause Analysis:KSAT, SCH, & PER - Login Issues This report outlines the findings and mitigations related to a service degradation that affected the KnowBe4 Security Awareness Training \(KSAT\), Security Coach \(SCH\), and PhishER \(PER\) platforms on **October 14, 2025**. The degradation was caused by an interaction between a recently deployed software update and the system’s internal connection management layer, which handles active session routing and authentication requests. The update introduced an inefficient database operation that caused high connection usage and resource exhaustion. Combined with inconsistent configuration parameters across service components, the system experienced widespread delays and login failures in multiple regions. The issue was reported beginning **1:00 p.m. UTC**, escalated through multiple channels, and was fully resolved by **6:45 p.m. UTC** after the software update was rolled back and the affected connection manager was replaced. # WHAT HAPPENED The KSAT platform manages user authentication and campaign scheduling across a distributed architecture that balances thousands of concurrent sessions in the impacted regions \(U.S. and E.U.\). On **October 14 at 1:00 p.m. UTC**, customers began to experience login delays and timeouts. Internal monitoring also alerted that session handling services were under heavy load and that response times had degraded significantly. Initial investigations pointed to abnormal resource consumption in database operations. The team later traced the problem to a software update deployed earlier in the day that contained an **inefficient database operation** executed at high frequency. This change sharply increased the number of active connections and caused spikes in backend resource utilization, resulting in authentication timeouts and intermittent login failures. # ROOT CAUSE ## Primary Root Cause: Inefficient Database Operation * **Resource Overload:** The newly deployed update introduced a repetitive operation that consumed disproportionate database and connection resources under normal user load. * **Insufficient Load Testing:** The operation passed standard validation but had not been tested under full production-scale concurrency, allowing the degradation to manifest only post-deployment. ## Secondary Root Cause: Connection Pool Fragmentation * **Configuration Inconsistency:** Varying connection parameters across services caused the connection manager to open independent resource pools, reducing efficiency and preventing resource reuse. * **Stale Resource Retention:** Locked pools held unused connections beyond their intended lifespan, contributing to system-wide slowdowns. * **Limited Observability:** Lack of detailed logging and automated recovery mechanisms delayed detection and response. # FINDINGS AND MITIGATIONS ## 1. Inefficient Database Operation **Finding:** The operation deployed on October 14 led to rapid degradation within minutes under production traffic.**Mitigations:** * **2:20 p.m. UTC:** Rolled back the software update, immediately stabilizing performance in the E.U. region. * Documented and removed the problematic query path for future validation and regression testing. ## 2. Connection Manager Fragmentation **Finding:** Inconsistent configuration parameters caused isolated pools and prevented efficient resource reuse.**Mitigations:** * **6:45 p.m. UTC:** Deployed a replacement connection management service, restoring performance in the U.S. region. * Enabled enhanced diagnostic logging for connection lifecycle events. * Established a verified escalation path for recycling resource pools if similar symptoms appear. ## 3. Configuration Standardization **Finding:** Non-standardized connection settings across components led to fragmentation.**Mitigations:** * Initiated engineering tasks to standardize connection parameters across all services. * Introduced a single configuration pattern per authentication context, with clear exceptions only for read/write separation. ## 4. Observability Improvements **Finding:** Limited insight into connection pool utilization delayed identification of the root cause.**Mitigations:** * Enhanced monitoring to surface anomalies in active connection counts and pool usage. * Documented the connection management architecture and incident playbook for on-call teams. # TECHNICAL DETAILS The database connection layer distributes incoming requests through multiple service nodes, each maintaining a limited set of reusable connection pools. When the inefficient operation was deployed, it introduced a unique connection pattern inconsistent with the rest of the application. This caused the manager to allocate new, isolated pools across each node, instead of reusing existing ones. As a result, thousands of connections remained tied up within these fragmented pools and could not be recycled by other services. Although sufficient overall database capacity remained, the platform’s session routing logic was unable to access it efficiently, resulting in high latency and login failures. Replacing the connection manager and reverting the software update ultimately allowed the release of resources and restored normal throughput. ## **PREVENTIVE MEASURES** * **Configuration Standardization:** All services will adhere to a unified connection configuration model to prevent fragmentation. * **Expanded Load Testing:** New performance tests simulate realistic traffic and concurrency to detect inefficiencies pre-deployment. * **Improved Connection Lifecycle Controls:** Standardized procedures now exist for resetting connection pools during active incidents. * **Documentation and Training:** Engineering teams have completed updated documentation for connection handling and troubleshooting. * **Enhanced Monitoring:** System metrics now include detailed connection pool utilization and latency correlation across nodes. ## **CUSTOMER IMPACT AND RECOVERY** Between **1:00 p.m. and 6:45 p.m. UTC**, some customers in the U.S. and E.U. regions experienced login failures and performance delays when accessing KSAT, SCH, or PER. Following the **rollback at 2:20 p.m.** and **connection manager replacement at 6:45 p.m.**, all regions stabilized and users regained normal access. No data loss occurred, though time-sensitive operations such as authentication, campaign delivery, and training notifications were delayed during the incident window. ## **CONCLUSION** This incident underscored the importance of consistent configuration management and comprehensive load testing for high-traffic authentication systems. The combination of an inefficient operation and fragmented connection handling led to resource exhaustion and widespread login disruptions. As a result of this RCA: * The faulty update has been permanently reverted. * The connection management service has been modernized and redeployed. * Standardization and observability improvements are underway across engineering teams. KnowBe4 remains committed to maintaining **high availability, transparency, and reliability** across all KSAT, SCH, and PER environments, and continues to invest in proactive safeguards to prevent recurrence.

Read the full incident report →

Started Oct 13, 2025, 07:50 PM UTC· resolved Oct 14, 2025, 11:37 AM UTC

1. investigating Oct 13, 2025, 07:50 PM UTC

   Customers may be unable to log in to their KSAT and PhishER console in the CA instance. We are investigating this issue and will update this page when we have more information.

2. investigating Oct 13, 2025, 08:30 PM UTC

   We have received reports that this is intermittently impacting all instances of KSAT and PhishER. You may have issues logging in and accessing your console. Our team is continuing to investigate this issue, and we will update this page as soon as we have more information.

3. monitoring Oct 13, 2025, 10:00 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Oct 13, 2025, 10:07 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Oct 14, 2025, 11:37 AM UTC

   This incident has been resolved.

6. postmortem Oct 27, 2025, 02:15 PM UTC

   # **External Technical Root Cause Analysis:KSAT & PhishER Login Failures – Canada Environment** This report outlines the findings and mitigations related to a login outage affecting the KnowBe4 Security Awareness Training \(KSAT\) and PhishER platforms within the Canada \(CA\) environment. The issue originated from a recently deployed data processing workflow that interacted unexpectedly with internal database components responsible for authentication and enrollment. The resulting database locks caused complete login failures for all users in the Canada region between **7:30 p.m. and 9:58 p.m. UTC on October 13, 2025**. The incident was fully resolved following the removal of the data processing workflow, which immediately restored normal authentication operations. No data loss occurred. ## **WHAT HAPPENED** On **October 13, 2025 at 7:30 p.m. UTC**, engineers observed login errors in the KSAT Canada environment, quickly followed by similar failures in PhishER. Automated monitoring confirmed that authentication requests were timing out at the database layer. Initial hypotheses focused on campaign update backlogs and database saturation. However, further analysis revealed the issue coincided with the deployment of a new **data export workflow** earlier that same day. This workflow initiated intensive queries against shared authentication tables, resulting in locks that blocked concurrent login and enrollment operations. A secondary issue compounded the impact: a **configuration mismatch** in the workflow caused its exclusion filters to fail. These filters were intended to skip large records to reduce load, but due to a naming discrepancy in the production database schemas, they were inadvertently ignored. As a result, the workflow attempted to process all records, including oversized ones, triggering crashes and automatic restarts. This “crash loop” continuously re-acquired locks on critical tables, preventing recovery until the workflow was removed. ## **ROOT CAUSE** **Primary Root Cause:** Data Processing Workflow Deployment * **Database Lock Contention:** The new workflow issued heavy queries against authentication tables, creating locks that prevented user sessions and enrollment updates from completing. * **Timeout Cascade:** Under sustained load, these locks led to cascading authentication timeouts and full login outages. **Secondary Root Cause:** Configuration Mismatch and Crash Loop * **Failed Exclusions:** The workflow’s exclusion settings targeted outdated schema names, invalidating safeguards designed to prevent oversized record processing. * **Crash and Restart Behavior:** When encountering unprocessable records, the workflow automatically restarted, repeatedly locking the same tables and prolonging the outage. ## **FINDINGS AND MITIGATIONS** 1. Workflow Removal and Database RecoveryFinding: The data processing workflow introduced locking contention on production authentication tables.**Mitigation:** * **Oct 13, 9:58 p.m. UTC:** Workflow fully removed from CA environment. * Database locks released immediately; login operations restored. * Post-removal monitoring confirmed that there was no residual impact on enrollment or authentication services. 1. Application Reconfiguration and VerificationFinding: Authentication timeouts persisted intermittently during investigation due to residual database contention.**Mitigation:** * Redeployed affected services to reinitialize connections. * Refactored export connector logic to prevent interaction with high-traffic authentication tables. 1. Observability and Isolation TestingFinding: The lack of targeted database lock monitoring delayed the identification of the blocking component.**Mitigation:** * Introduced component isolation tests to identify lock sources more quickly. * Added diagnostic telemetry for query lock patterns and workflow restart behavior. ## **TECHNICAL DETAILS** The deployed data export workflow executed read-heavy queries on shared user authentication and enrollment tables. These operations held **shared locks** incompatible with concurrent updates, blocking normal authentication processes. Due to the configuration mismatch, exclusion filters did not apply, allowing the workflow to process large binary data columns. Each crash triggered automatic restarts, which in turn retried the same operations—reacquiring locks and creating a repeating outage loop. Removal of the workflow immediately stopped lock contention, allowing the database to clear queued transactions and restore login functionality. ## **PREVENTIVE MEASURES** * **Environment Validation:** Align database naming conventions between test and production environments to ensure exclusion filters function as intended. * **Database Lock Monitoring:** Implement proactive monitoring and alerting for lock contention on critical tables. * **Workflow Design Safeguards:** Introduce alternative mechanisms to avoid live locks on transactional databases. ## **CUSTOMER IMPACT AND RECOVERY** Between **7:30 p.m. and 9:58 p.m. UTC**, customers in the **Canada \(CA\)** region experienced issues logging in to KSAT and PhishER. All other regions remained unaffected. Following workflow removal, services recovered immediately, and all users regained access. No data loss or corruption occurred. ## **CONCLUSION** This incident was caused by a combination of resource contention and configuration mismatch in a newly deployed workflow. The interplay between database locking and restart behavior created a self-sustaining failure loop that fully blocked authentication. As a result of this incident: * The problematic workflow has been removed from production. * Enhanced deployment validation and observability have been implemented. * Database lock monitoring and alternative extraction approaches are being deployed to prevent recurrence. KnowBe4 remains committed to ensuring **high availability, transparency, and reliability** across all KSAT and PhishER environments.

Read the full incident report →

Started Oct 10, 2025, 03:48 PM UTC· resolved Oct 10, 2025, 05:21 PM UTC

1. identified Oct 10, 2025, 03:48 PM UTC

   We've identified the cause of Training and the Modstore not loading and are working on implementing a fix. We'll continue to post on our status page with any new information or updates.

2. resolved Oct 10, 2025, 05:21 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 09, 2025, 09:18 AM UTC· resolved Oct 09, 2025, 02:34 PM UTC

1. investigating Oct 09, 2025, 09:18 AM UTC

   We are currently investigating this issue.

2. investigating Oct 09, 2025, 09:19 AM UTC

   We are continuing to investigate this issue.

3. investigating Oct 09, 2025, 09:23 AM UTC

   We are continuing to investigate this issue.

4. identified Oct 09, 2025, 10:52 AM UTC

   We’ve identified the issue as being related to Azure "Front Door" service outages in multiple regions. Our team is in active contact with Microsoft and working to mitigate the impact. We’ll continue to share updates here as more information becomes available.

5. monitoring Oct 09, 2025, 12:32 PM UTC

   We have seen improvements with Azure's "Front Door" service and we are monitoring the results to make sure no further issues occur.

6. monitoring Oct 09, 2025, 12:37 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Oct 09, 2025, 02:34 PM UTC

   This incident has been resolved.

Read the full incident report →