CyberQP Outage History

Started Oct 16, 2026, 08:00 PM UTC· resolved Oct 17, 2026, 01:42 PM UTC

1. resolved Oct 19, 2026, 04:31 PM UTC

   What happened: Starting around 20:50 UTC on October 16, 2026, some users in the US region couldn't sign in to Quickpass QTech chrome extension. They saw a "403 Forbidden" error, and some briefly saw a "Server Error, try again in 30 seconds" message. Over the next few days it affected a growing number of login attempts. Root cause: Our web application firewall uses a set of rules managed by our cloud provider. Those rules were updated on the provider's side, and one of them started wrongly flagging a normal part of our sign-in process as a threat. The firewall then blocked those login requests. We made no changes to our systems before the issue started. Security impact: There was no security breach, and no data was exposed or compromised. The firewall blocked legitimate traffic. Nothing got past it that shouldn't have. Our authentication systems were healthy throughout, and no customer or partner data was accessed. The only impact was that users couldn't log in. Resolution: We adjusted the firewall configuration so the sign-in process works again. All other protections stay fully in place. Logins were restored at 1:42 PM UTC, 19th Oct 2026.

2. postmortem Oct 19, 2026, 04:31 PM UTC

   What happened: Starting around 20:50 UTC on October 16, 2026, some users in the US region couldn't sign in to Quickpass QTech chrome extension. They saw a "403 Forbidden" error, and some briefly saw a "Server Error, try again in 30 seconds" message. Over the next few days it affected a growing number of login attempts. Root cause: Our web application firewall uses a set of rules managed by our cloud provider. Those rules were updated on the provider's side, and one of them started wrongly flagging a normal part of our sign-in process as a threat. The firewall then blocked those login requests. We made no changes to our systems before the issue started. Security impact: There was no security breach, and no data was exposed or compromised. The firewall blocked legitimate traffic. Nothing got past it that shouldn't have. Our authentication systems were healthy throughout, and no customer or partner data was accessed. The only impact was that users couldn't log in. Resolution: We adjusted the firewall configuration so the sign-in process works again. All other protections stay fully in place. Logins were restored at 1:42 PM UTC, 19th Oct 2026.

Read the full incident report →

Started Sep 03, 2026, 01:18 PM UTC· resolved Sep 03, 2026, 01:57 PM UTC

1. investigating Sep 03, 2026, 01:18 PM UTC

   We're experiencing an high priority issue with the Agent connectivity, and are currently looking into the root cause. Apologias for the inconvenience. All updates will be posted here.

2. identified Sep 03, 2026, 01:25 PM UTC

   The issue has been identified and addressed

3. monitoring Sep 03, 2026, 01:27 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 03, 2026, 01:57 PM UTC

   This incident has been resolved.

5. postmortem Sep 03, 2026, 07:27 PM UTC

   ## Service Incident Report — EU Region **Date:** 3 September 2026 **Duration:** 11:49 – 13:23 UTC \(1 hour 34 minutes\) **Region affected:** EU **Status:** Resolved ### Summary On 3 September 2026, our agent connectivity service in the EU region became unavailable for approximately 1 hour 34 minutes. During this window, agents in the EU region were unable to establish connections, and scheduled background job processing was delayed. Service was fully restored by 13:23 UTC and has been stable since. Regions other than EU were unaffected. ### Impact For the duration of the incident, in the EU region only: * **Agents could not connect** to the platform. Existing sessions were interrupted and new connections failed. * **Scheduled and queued background jobs were delayed.** Queued work resumed processing after recovery. ### What happened Two factors combined: A memory management issue in our agent connectivity service caused it to gradually consume increasing memory over an extended runtime, eventually making it slow to respond to automated health checks. Separately, the health check timing configuration in our EU region allowed a narrower recovery window than in other regions. As a result, instances that would otherwise have recovered were restarted automatically before they could return to service, and those restarts escalated into a repeated restart cycle rather than self-healing. An automated recovery process then replaced the affected instances, but the replacements encountered the same timing constraint on startup, which extended the outage rather than ending it. ### Resolution Our engineering team identified the cause and applied a configuration correction directly to the EU environment at 13:23 UTC, aligning its health check timing and resource allocation with values already proven in our other regions. Both affected services returned to a healthy state immediately and ran for over 100 minutes with zero restarts before the incident was closed. ### Preventive actions 1. **Health check timing corrected in the EU region** — completed 3 September 2026. 2. **Proactive review and alignment of the same configuration across all remaining regions**, so no region carries a narrower recovery margin than another. 3. **New automated alerting on repeated service restarts**, so this class of degradation is detected and escalated before it affects availability. 4. **A safeguard added to our automated recovery process** to prevent it from restarting all instances of a service simultaneously. 5. **Continued engineering work to eliminate the underlying memory management issue** in the agent connectivity service, rather than relying on tolerance settings alone. ### Questions For questions about this incident or its impact on your tenants, contact [[email protected]](mailto:[email protected]).

Read the full incident report →

Started Apr 07, 2026, 07:30 PM UTC· resolved Apr 13, 2026, 10:33 PM UTC

1. investigating Apr 07, 2026, 07:30 PM UTC

   We are currently investigating reports of slowness affecting the CyberQP dashboard. Our team has been made aware of the issue and investigation is underway. We will provide further updates as more information becomes available.

2. identified Apr 07, 2026, 08:22 PM UTC

   We have identified the cause of the Dashboard slowness and are working on mitigation.

3. identified Apr 08, 2026, 10:23 PM UTC

   Audit events performance is currently degraded. Our team is actively investigating and working to resolve the issue as quickly as possible.

4. monitoring Apr 09, 2026, 06:32 PM UTC

   We have deployed a fix, and everything is operating normally. Our team will continue to monitor closely to ensure stability.

5. resolved Apr 13, 2026, 10:33 PM UTC

   After monitoring over the weekend, we're seeing normal Dashboard responsiveness. This incident has been resolved.

Read the full incident report →

Started Feb 26, 2026, 10:57 PM UTC· resolved Feb 27, 2026, 01:18 AM UTC

1. investigating Feb 26, 2026, 10:57 PM UTC

   We are currently investigating an issue affecting Entra/M365 Just in Time (JIT) accounts that have been re-enabled after a period of being disabled. In this state, the password displayed by the Dashboard and Chrome Extension does not match the active credential, preventing successful login. Active Directory and Local JIT accounts are not affected. New Entra/M365 JIT account creation also continues to function normally. Workaround: Deleting the affected JIT account and recreating it from scratch will resolve the issue. Please note this will require the end user to re-register MFA. We recommend evaluating the impact of MFA re-registration in your environment before proceeding. Our team is treating this as the highest priority and is actively working toward a resolution.

2. identified Feb 27, 2026, 12:20 AM UTC

   We have identified the likely cause: a synchronization delay on the Microsoft Entra/M365 side. Waiting approximately 5 minutes after re-enabling a Just in Time account will allow login with the supplied password. We are continuing to investigate whether this delay can be reduced on our end, or whether it is inherent to Microsoft's platform. Further updates to follow.

3. resolved Feb 27, 2026, 01:18 AM UTC

   We’ve completed our review of the reported login delays impacting Just-in-Time (JIT) accounts. Our investigation, including comparison of our activity logs with Microsoft Entra Audit Logs, confirms that credentials are being generated and delivered correctly at the time of activation. The delay is occurring between the password reset event and when the updated credentials become active within Entra/M365. We are continuing to monitor this behavior and will provide a further update once latency returns to expected levels.

4. postmortem Mar 03, 2026, 07:31 PM UTC

   Although the root cause was not within our platform, the impact was material and we recognize the additional operational overhead this created for your team. We have now released an enhancement to the Microsoft Entra ID / Microsoft 365 Just in Time \(JIT\) integration that successfully works around the issue observed beginning February 26 at approximately 12:00 PM EST. **What This Means for You** ‌ **Currently Enabled Entra/M365 JIT Accounts** * No action is required. These accounts will function normally moving forward. ‌ **Currently Disabled Entra/M365 JIT Accounts** Action is required: * Re-enable the JIT account. * Using either the Chrome extension or the QTech Mobile application, extend the JIT duration. After this step is completed, all subsequent disable/re-enable cycles for that account will function normally without additional intervention. **Alternative Option** You may delete and recreate the JIT account. However, this will require the user to re-register MFA and may introduce additional administrative effort and delay.

Read the full incident report →

Started Aug 11, 2025, 05:18 PM UTC· resolved Aug 11, 2025, 07:08 PM UTC

1. investigating Aug 11, 2025, 05:18 PM UTC

   We are aware of a bug causing Just in Time Activation and Creation to present a message that no Just in Time Policy are available and an "invalidCustomerID" error banner appearing at the top of the page. This has been escalated to the highest priority with the Development Team and will be addressed as quickly as possible. We will notify via any existing Support Tickets and via the Status Page when the resolution has been released. We apologize for the inconvenience this has caused.

2. identified Aug 11, 2025, 06:01 PM UTC

   The root cause of the incident has been discovered and our team is working to quickly release a fix to resolve this. We will update again when we have released that resolution.

3. monitoring Aug 11, 2025, 07:01 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 11, 2025, 07:08 PM UTC

   This incident has been resolved. Please ensure that a full refresh of the Dashboard page is attempted prior to testing again. (CTRL F5 or the equivalent)

Read the full incident report →

Started Jun 12, 2025, 07:50 PM UTC· resolved Jun 12, 2025, 10:28 PM UTC

1. investigating Jun 12, 2025, 07:50 PM UTC

   Due to the outage with Google GCP Services, many functions of our solution are currently inaccessible. We are closely monitoring the Google GCP Outage and will advise of the status as the GCP Services come back online.

2. monitoring Jun 12, 2025, 09:13 PM UTC

   As Google GCP Services return to functional status, we're starting to see a return to service, however not all functionality has been returned. We will continue to update as we monitor.

3. monitoring Jun 12, 2025, 09:13 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Jun 12, 2025, 10:28 PM UTC

   Google Cloud Platform (GCP) services are now reported to be nearly fully operational, with final recovery steps expected to conclude shortly. Based on internal validation, all Dashboard components and associated functionalities have been restored and are functioning within normal parameters. If you continue to encounter any anomalies or degraded performance, please respond to this email for further investigation.

Read the full incident report →

Started Feb 24, 2025, 09:52 PM UTC· resolved Feb 24, 2025, 10:51 PM UTC

1. investigating Feb 24, 2025, 09:52 PM UTC

   We’ve identified an issue from a recent release affecting JIT account enablement (AD/Local/EntraID) via the Admin Dashboard and Passwordless authentication. JIT account creation is unaffected, but enabling pre-existing JIT accounts is not working as expected. Our team is actively working on a fix, and we expect full functionality to be restored shortly.

2. resolved Feb 24, 2025, 10:51 PM UTC

   The issue affecting JIT account enablement (AD/Local/EntraID) via the Admin Dashboard and Passwordless authentication has been successfully resolved. All functionality is now fully restored. Please reach out if you encounter any further issues.

Read the full incident report →

Started Feb 14, 2025, 05:41 PM UTC· resolved Feb 14, 2025, 08:53 PM UTC

1. monitoring Feb 14, 2025, 05:41 PM UTC

   We have received reports regarding outbound emails from our solution being queued in our email delivery system. After thorough investigation, we have identified the cause of the issue in collaboration with our email delivery vendor. As a result, you may experience a bulk delivery of emails or receive notifications that were previously sent out. We apologize for any inconvenience this may cause and want to assure you that we are actively monitoring the situation. We will provide an update once the queue has been cleared. Thank you for your understanding and patience.

2. monitoring Feb 14, 2025, 06:33 PM UTC

   We have started to see the queue of backlogged emails processing now. You should start to see emails arriving in your designated inbox. We will continue to monitor and advise when the entire backlog has been processed.

3. resolved Feb 14, 2025, 08:53 PM UTC

   We see the queue of backlogged emails has cleared. Please reply to this email if you are still seeing significant delays on email delivery.

Read the full incident report →

Started Jan 26, 2025, 06:06 PM UTC· resolved Jan 26, 2025, 07:37 PM UTC

1. investigating Jan 26, 2025, 06:06 PM UTC

   We have begun investigating the possible cause. This is only affecting the EU Admin Dashboard at this time.

2. monitoring Jan 26, 2025, 06:24 PM UTC

   We have isolated and resolved the cause of the login challenge. EU Logins now are working. Please reply to this email if you are still having challenges or open a ticket by sending an email to [email protected]

3. resolved Jan 26, 2025, 07:37 PM UTC

   With no further reports of login issues, this incident is being marked as Resolved.

Read the full incident report →

Started Jan 10, 2025, 03:26 PM UTC· resolved Jan 10, 2025, 10:45 PM UTC

1. investigating Jan 10, 2025, 03:26 PM UTC

   We are aware that Agent version 5.3.6.0 update is being flagged as potentially malicious by Webroot. PLEASE NOTE: We can confirm this is a FALSE POSITIVE. If you are impacted, kindly submit this to Webroot as a FALSE POSITIVE. Our security and development teams are actively investigating the issue and working to address why this update was flagged. We’ll share updates as soon as more information becomes available.

2. monitoring Jan 10, 2025, 07:34 PM UTC

   A fix has been implemented and we are monitoring the results.

3. investigating Jan 10, 2025, 07:35 PM UTC

   We are aware that Agent version 5.3.6.0 update is being flagged as potentially malicious by Webroot. PLEASE NOTE: We can confirm this is a FALSE POSITIVE. If you are impacted, kindly submit this to Webroot as a FALSE POSITIVE. Our security and development teams are actively investigating the issue and working to address why this update was flagged. We’ll share updates as soon as more information becomes available.

4. resolved Jan 10, 2025, 10:45 PM UTC

   Webroot support indicates that they have whitelisted the Agent update for 5.3.6.0 - we are hopeful their support team will be willing to work with us for future Agent updates so that this doesn't happen again. Agent updates, that failed, may require the Agent to be reinstalled. You can simply install over top of the existing installation via your RMM, scripting process, or manually via the GUI.

Read the full incident report →

Started Dec 12, 2024, 07:02 PM UTC· resolved Dec 12, 2024, 07:34 PM UTC

1. investigating Dec 12, 2024, 07:02 PM UTC

   We are seeing reports of an error message banner appearing when attempting to access the Customer List on the Dashboard. "Invalid `prisma.$executeRaw()` invocation: Authentication failed against database server at ..........etc." Our team has already begun the investigation and we'll update you ASAP on the resolution/fix.

2. investigating Dec 12, 2024, 07:25 PM UTC

   We are continuing to investigate this challenge. We have isolated this to just Tenant that use our Vault. All other Customers with ITGlue or Hudu integrations do not seem to be affected. This is the top priority for our team to get resolved.

3. investigating Dec 12, 2024, 07:32 PM UTC

   We are continuing to investigate this issue.

4. resolved Dec 12, 2024, 07:34 PM UTC

   The resolution of the challenge for access to the Vault based Tenants has been resolved. The development team was able to quickly identify and remediate this challenge. Please do a full refresh of the Dashboard (CTRL - F5 on Chrome based browsers) and access your customers. If you are still seeing this error message, please reply to this email and a ticket will be created for you.

Read the full incident report →

Started Oct 30, 2024, 03:47 PM UTC· resolved Oct 30, 2024, 07:43 PM UTC

1. investigating Oct 30, 2024, 03:47 PM UTC

   We are currently investigating an Error message that appears when creating new Just in Time (JIT) accounts. We will continue to update this incident with our progress on resolution.

2. identified Oct 30, 2024, 04:40 PM UTC

   We have identified the cause of this challenge and the Development team is testing in our QA environment. Once we confirm this and release it to the public we'll send another update.

3. monitoring Oct 30, 2024, 06:13 PM UTC

   A solution has been successfully implemented to resolve the issue preventing the creation of Just-in-Time (JIT) accounts. This fix has been deployed to the production environment, and active monitoring is in place to ensure continued performance and stability. Further updates will be provided as needed.

4. resolved Oct 30, 2024, 07:43 PM UTC

   Due to positive feedback from our MSP Partners, this incident will be closed. If you are still seeing any challenge related to this, please reply to this email and a ticket will be created for you.

Read the full incident report →