New Modstore 500 Errors

Update timeline

1. identified May 21, 2026, 01:40 PM UTC

   We've received reports that customers are unable to access the new Modstore experience and are receiving 500 errors. We are investigating these issues and will update this page when we have more information.

2. monitoring May 21, 2026, 01:41 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 21, 2026, 05:20 PM UTC

   This incident has been resolved.

4. postmortem Jul 07, 2026, 03:32 PM UTC

   # **Summary** On Wednesday, May 21, 2026, customers using the new ModStore experience within KnowBe4 Security Awareness Training \(KSAT\) were unable to access the ModStore and received HTTP 500 errors. The disruption affected all regional instances and lasted approximately 23 minutes, from 13:17 to 13:40 \(UTC\). This issue was caused by a defective code change in a routine ModStore deployment. The deployment itself completed successfully, but the change caused the application to return server errors when customers attempted to load the new ModStore. Our engineering team identified the faulty deployment within minutes, rolled it back, and confirmed full restoration of service at 13:40 \(UTC\). Only the new ModStore experience was affected. Customers using the classic ModStore, and all other KSAT functionality — including training assignments, campaigns, and reporting — remained fully operational throughout. There was no data loss as a result of this incident. # **What Happened** ## **New ModStore \(KSAT\)** At 13:08 \(UTC\) on May 21, 2026, a routine deployment of the new ModStore application began and was completed successfully across all production environments at 13:17 \(UTC\). The release included changes related to integrating the new ModStore natively into the KnowBe4 platform. Immediately following the deployment, the new ModStore began returning HTTP 500 errors to customers attempting to access it. At 13:28 \(UTC\), internal reports of the errors reached the engineering team, and by 13:30 \(UTC\) engineers had correlated the failures with the deployment. The decision to roll back was made at 13:31 \(UTC\). At 13:32 \(UTC\), with multiple customer support tickets confirming customer-facing impact, a high-severity incident was declared, and on-call engineers were contacted. The team quickly confirmed the scope: only the new ModStore experience, enabled for approximately half of customers plus those who had opted in, was affected, across all regional instances. The classic ModStore and all other KSAT functionality remained available. A rollback to the previous stable version was initiated at 13:35 \(UTC\) and completed at 13:40 \(UTC\), at which point access to the new ModStore was fully restored. Our public status page was updated at 13:40 \(UTC\) and moved to “Monitoring” one minute later. During the post-restoration investigation, an engineer noted that a separate infrastructure-as-code deployment earlier that morning had unexpectedly altered a storage Cross-Origin Resource Sharing \(CORS\) configuration used by the ModStore. This change was investigated as a potential contributor and ruled out — it was not related to the 500 errors. The investigation did, however, surface that two separate deployment pipelines were both managing the same infrastructure configuration and silently overwriting each other’s changes. This conflict was remediated during the incident window by assigning the configuration a single owning pipeline. After continued monitoring confirmed stability, the incident was resolved internally at 14:52 \(UTC\), and the public status page was updated to “Resolved” at 17:20 \(UTC\). # **Root Cause Analysis** The root cause of this incident was a defective code change included in the 13:17 \(UTC\) deployment of the new ModStore application. Once deployed, the change caused the application to fail to serve customer requests, returning HTTP 500 errors to all users of the new ModStore experience across all regions. The defect was not detected during pre-deployment testing, so the deployment proceeded to production as usual. Because the failure began at the moment the deployment completed and affected all regions simultaneously, engineers were able to identify the deployment as the trigger within minutes and end customer impact by rolling back to the previous version. The faulty change was withheld for rework and additional validation before any reintroduction. A secondary issue was identified during the investigation: an unrelated infrastructure configuration change that morning initially appeared connected because of its timing. It was ruled out as a cause, but the investigation revealed that two deployment pipelines shared ownership of the same infrastructure configuration and were overwriting each other. While this did not cause the incident, it added noise to the diagnosis. Because it represented a latent risk, it was also corrected the same day. ## **Detailed Timeline \(UTC\)** | **Time \(UTC\)** | **Event** | | --- | --- | | **13:08** | A routine deployment to the new ModStore application begins. | | **13:17** | The deployment completes across all production environments; HTTP 500 errors begin for the new ModStore experience. | | **13:28** | Internal reports of errors loading the new ModStore reach the engineering team. | | **13:30** | Engineers correlate the HTTP 500 errors with the earlier deployment. | | **13:31** | Decision made to roll back the deployment. | | **13:32** | A high-severity incident is declared; on-call engineers are paged. Customer support tickets confirm customer-facing impact. | | **13:33 – 13:34** | Impact confirmed to be limited to the new ModStore experience; the classic ModStore and all other KSAT functionality are confirmed unaffected. All regional instances are affected. | | **13:35 – 13:38** | The revert is prepared, and the rollback deployment begins. | | **13:40** | Rollback completes, and access to the new ModStore is restored. The public status page is updated to “Identified.” | | **13:41** | The public status page is updated to “Monitoring.” | | **13:58 – 14:11** | An unrelated same-morning infrastructure configuration change is investigated and ruled out as a cause. A conflicting ownership issue between two deployment pipelines that manage the same configuration is identified and remediated. | | **14:52** | After continued monitoring confirms stability, the incident is marked resolved internally \(74 minutes after the incident was declared\). | | **17:20** | The public status page is updated to “Resolved.” | ‌ # **Findings and Mitigations** ## **1. A defective change reached production** A code change included in a routine deployment of the new ModStore caused the application to return server errors in production. The defect was not caught by the automated tests that run before a release is promoted, indicating a gap in pre-deployment test coverage for this failure mode. **Mitigations:** * The deployment was rolled back within 23 minutes of impact beginning, immediately restoring service. * The faulty change was withheld from redeployment pending rework and additional validation. * Pre-deployment test coverage for the new ModStore is being expanded to cover the failure mode seen in this incident. ## **2. Detection relied on human reports rather than automated alerting** Impact began at 13:17 \(UTC\), but the engineering team was first engaged through internal reports at 13:28 \(UTC\) and customer support tickets shortly after, rather than by an automated alert on the application’s error rate. Automated post-deployment checks that ran after the release did not halt the rollout or page the team. **Mitigations:** * Error-rate monitoring and automated alerting for the new ModStore are being strengthened so that a spike in server errors immediately after a deployment pages the on-call team directly. * Post-deployment verification is being reviewed so that failing checks more decisively block or flag a release. ## **3. Two deployment pipelines managed the same infrastructure configuration** The application’s deployment pipeline and a separate infrastructure-as-code pipeline both defined the same storage CORS configuration, and each deployment silently overwrote the other’s settings. This conflict was not the cause of the incident, but it initially complicated diagnosis and represented an ongoing risk of unintended configuration changes. **Mitigations:** * The duplicated configuration was removed from the application pipeline and its state references were cleaned up, making the infrastructure pipeline the single owner of that configuration. * A review is underway to identify any other resources with shared ownership across pipelines. # **Customer Impact** ## **KSAT – New ModStore \(all regions\)** From 13:17 to 13:40 \(UTC\), customers who had the new ModStore experience enabled \(approximately half of customers, plus those who opted in\) received HTTP 500 errors when attempting to access the ModStore and were unable to browse or add training content during that window. All regional instances were affected equally. Customers using the classic ModStore were not affected. All other KSAT functionality — including active training campaigns, phishing simulations, user enrollments, and reporting — operated normally throughout the incident. No customer action was or is required, and no data was lost or altered. # **Preventive Measures** * **Expanded pre-deployment testing —** Test coverage for the new ModStore is being extended to catch the class of defect that caused this incident before a release reaches production. * **Automated error-rate alerting —** Monitoring on the new ModStore is being strengthened so that elevated server-error rates following a deployment automatically notify on-call engineers, removing the dependency on human reports for detection. * **Stronger post-deployment verification —** The automated checks that run immediately after a release are being reviewed so that failures block or escalate a rollout rather than passing silently. * **Single ownership of infrastructure configuration —** Shared infrastructure settings are being consolidated under a single owning pipeline to prevent conflicting automated changes, and an audit is underway to identify any remaining overlaps. # **Conclusion** On May 21, 2026, a defective code change during a routine deployment made the new ModStore experience unavailable for approximately 23 minutes, resulting in server errors for affected customers across all regions. We recognize that the ModStore is central to how administrators build their training programs, and we apologize for the disruption. The response demonstrated the value of fast rollback as a recovery path: the faulty release was identified and reverted within minutes of the first reports, the scope was accurately confirmed early, and no data was lost. The incident also highlighted clear opportunities to improve: stronger pre-deployment testing, automated error-rate detection, and cleaner ownership of infrastructure configuration. These are being addressed through the preventive measures above. We are committed to the reliability of the new ModStore experience as it rolls out to all customers, to ensuring that issues of this kind are caught before they reach production, and to detecting and resolving issues automatically if they do reach production.