Vasion Outage History

Started Feb 10, 2026, 05:42 AM UTC· resolved Feb 12, 2026, 08:40 PM UTC

1. investigating Feb 10, 2026, 05:42 AM UTC

   For customers based in EMEA and APAC regions, we are aware of a potential issue impacting those who utilize the Control Panel Application (CPA) on Ricoh devices. Customers with auto updates enabled, or those manually updating Ricoh CPAs, may experience degraded or interrupted performance. We are working to mitigate further impact.

2. identified Feb 10, 2026, 09:58 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Feb 11, 2026, 02:26 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Feb 11, 2026, 02:27 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Feb 11, 2026, 11:48 PM UTC

   A fix has been implemented in our APAC and EMEA regions. Fix for North America region will be in the maintenance release planned for next week

6. resolved Feb 12, 2026, 08:40 PM UTC

   This incident has been resolved.

7. postmortem Feb 16, 2026, 12:42 AM UTC

   **Ricoh Control Panel Application Service Disruption** ‌ **Issue Summary:** On February 9, 2026, the Control Panel Application \(CPA\) failed to come back online following a scheduled service update, resulting in a loss of CPA functionality on affected Ricoh multifunction printers. Users were unable to access scan-to-email, scan-to-network, and other embedded features. **Root Cause:** The scheduled service update included an upgrade to the Ricoh SDK which required reinstallation of the CPA on all devices. However, the Ricoh-provided access token, which is necessary for the automated reinstallation process, expired on February 9, 2026, coinciding with the deployment window. Without a valid access token, the automated reinstallation process failed, leaving the CPA in a non-functional state. **Resolution:** Our engineering team immediately contacted Ricoh to request token renewal. Once the new access token was provisioned, we generated an emergency patch to deploy the fix to all customers. This patch restored CPA service to affected printers after deployment was completed. **Mitigation:** To prevent recurrence of this issue, the Ricoh access token expiration date has been documented in our maintenance calendar with automated alerts set for 30 days and 7 days prior to expiration. This will ensure tokens are renewed proactively and that similar authentication-related disruptions are identified prior to any production updates. **Conclusion:** We understand that the loss of CPA functionality significantly impacts daily operations, and we sincerely apologize for this service disruption. This incident has highlighted gaps in our credential lifecycle management process. The preventative measures outlined above will ensure proactive token renewal going forward. We remain committed to continuous improvement of our printer integration services and delivering the reliable performance you expect from Vasion.

Read the full incident report →

Started Feb 09, 2026, 02:03 AM UTC· resolved Feb 09, 2026, 04:02 AM UTC

1. investigating Feb 09, 2026, 02:03 AM UTC

   We are currently investigating this issue.

2. investigating Feb 09, 2026, 02:31 AM UTC

   We are continuing to investigate this issue.

3. monitoring Feb 09, 2026, 02:42 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 09, 2026, 04:02 AM UTC

   This incident has been resolved.

5. postmortem Feb 11, 2026, 09:36 PM UTC

   **Service Interruption** ‌ **Issue Summary:** On 08 Feb 2025 between 23:30 UTC and 09 Feb 2025 02:00 UTC \(4:30 PM MT - 7:00 PM MT\), the Vasion platform experienced intermittent service disruptions affecting a small subset of customers due to unusually high traffic volumes. ‌ **Root Cause:** The platform encountered an unexpected surge in traffic that exceeded normal operating capacity. This volume of requests overwhelmed specific components of our infrastructure, resulting in intermittent slowdowns and brief service outages for a limited number of affected customers. ‌ **Resolution:** Our operations team quickly identified the source of the elevated traffic and blocked abusive IPs to restore normal service levels. The service was fully restored by 02:00 UTC \(7:00 PM MT\). ‌ **Mitigation:** To prevent similar disruptions in the future, we are enhancing traffic monitoring and rate-limiting controls. These measures will enable our systems to detect and respond to abnormal traffic patterns more quickly, ensuring continued service availability even during unexpected traffic spikes. **‌** **Conclusion:** We recognize the critical importance of uninterrupted service to your business and sincerely apologize for any inconvenience this disruption caused to the affected customers. We are committed to maintaining the highest levels of service reliability and have taken concrete steps to strengthen our infrastructure's resilience against similar events in the future.

Read the full incident report →

Started Dec 17, 2025, 03:27 PM UTC· resolved Dec 17, 2025, 07:15 PM UTC

1. investigating Dec 17, 2025, 03:27 PM UTC

   We are currently investigating this issue.

2. identified Dec 17, 2025, 04:28 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Dec 17, 2025, 05:25 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Dec 17, 2025, 06:14 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Dec 17, 2025, 07:15 PM UTC

   This incident has been resolved.

6. postmortem Dec 19, 2025, 07:10 PM UTC

   **US - Bad Network Gateway interruption** ‌ **Issue Summary:** On December 17th, from 8:00 AM MST to 12:15 PM MST, all services requiring access to our hosted environment in the US region \([printercloud.com](http://printercloud.com)\) were unavailable, returning a 500 Bad Gateway error. Local services not requiring connections to our systems remained functional. The outage was caused by exhausted system resources due to a system package upgrade, preventing our systems from processing incoming requests. ‌ **Root Cause:** During routine system patching, the containerd package was updated, introducing breaking changes to resource management that caused cascading service failures across production environments. Services exhausted available system resources and could no longer accept network connections, causing the platform to become unstable. ‌ **Resolution:** While troubleshooting, the operations team applied a configuration override to restore appropriate system resource limits. Once the offending update was identified, the operations team rolled back the system-level deployment package. ‌ **Mitigation:** We are implementing staged deployment processes for system updates with mandatory testing in non-production environments, and expanding monitoring to alert on system configuration changes before potential impact to production services. As a next step, our Operations team will decouple OS-level security patches and package updates from Schedule Release SaaS application releases. This will allow us to: ‌ * Maintain our security posture with timely OS patches * Reduce the complexity and risk profile of application deployments * Improve our ability to identify root causes quickly if issues occur * Enable faster rollbacks if problems arise **‌** **Conclusion:** We acknowledge the impact this had on customers in the US region. We are committed to improving our processes to prevent future recurrence. Thank you for your understanding as our teams worked to resolve the issues stemming from this container package upgrade.

Read the full incident report →

Started Oct 17, 2025, 04:28 PM UTC· resolved Nov 06, 2025, 08:32 PM UTC

1. investigating Oct 17, 2025, 04:28 PM UTC

   We are currently investigating this issue.

2. identified Oct 21, 2025, 02:29 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Nov 04, 2025, 06:15 PM UTC

   The MSP Portal team has identified the root cause and has a fix is planned to be included in the next upcoming release

4. monitoring Nov 06, 2025, 04:31 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Nov 06, 2025, 08:32 PM UTC

   This incident has been resolved.

6. postmortem Nov 12, 2025, 05:55 PM UTC

   November 12, 2025 **MSP Portal: Instance creation hanging** ‌ **Issue Summary:** On October 17, 2025, some MSP partners experienced failures when attempting to create new customer instances through the MSP Portal. ‌ **Root Cause:** An internal data alignment issue caused our provisioning service to reject valid instance creation requests. Additionally, overly strict data validation rules incorrectly flagged some legitimate requests as invalid. ‌ **Resolution:** Our engineering team corrected the data alignment issue and implemented improved error handling to ensure the provisioning service processes instance creation requests successfully. ‌ **Mitigation:** We've adjusted the data validation rules to prevent false rejections while maintaining appropriate security standards. We've also added enhanced monitoring to detect similar issues before they impact customers. We also added better phone number validation that properly supports international numbers. **‌** **Conclusion:** This issue has been fully resolved, and the additional safeguards are now in place to prevent recurrence.

Read the full incident report →

Started Oct 13, 2025, 06:11 PM UTC· resolved Oct 13, 2025, 06:53 PM UTC

1. investigating Oct 13, 2025, 06:11 PM UTC

   We are currently investigating this issue.

2. resolved Oct 13, 2025, 06:53 PM UTC

   This incident has been resolved.

3. postmortem Oct 14, 2025, 02:50 PM UTC

   Monday, October 13th, 2025 ‌ **Intermittent Timeout Issues When Accessing Instance** ‌ **Issue Summary:** On 13 October 2025, between 5:13 PM UTC and 6:38 PM UTC, some users with instances hosted in printercloud6 experienced intermittent timeout or connection failure issues when accessing the platform. ‌ **Root Cause:** Multiple compute nodes became unresponsive, resulting in intermittent issues accessing other resources. ‌ **Resolution:** The affected compute nodes were identified and replaced. ‌ **Mitigation:** To address this issue going forward, additional monitors have been added. Our engineering teams’ investigation is still ongoing. Once further information is identified regarding what caused the nodes to become unresponsive, measures will be implemented to prevent this issue from occurring on compute nodes going forward. ‌ **Conclusion:** We acknowledge that this had an impact on some customers. We thank you for your patience as we resolved this service disruption.

Read the full incident report →

Started Sep 05, 2025, 12:57 PM UTC· resolved Sep 05, 2025, 03:42 PM UTC

1. investigating Sep 05, 2025, 12:57 PM UTC

   We are currently investigating this issue.

2. identified Sep 05, 2025, 01:29 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 05, 2025, 02:34 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 05, 2025, 03:42 PM UTC

   This incident has been resolved.

5. postmortem Sep 10, 2025, 03:51 PM UTC

   Friday, September 5, 2025 ‌ **Users experiencing issues authenticating from the printer \(badge, PIN\)** ‌ ‌ **Issue Summary:** Users experienced problems authenticating at printers using their badges or PINs. This prevented affected users from using all of their assigned badges for authentication at printers. ‌ ‌ **Root Cause:** During the latest SRS release, the synchronization process for user badges was modified, but the implementation only accounted for syncing a single badge per user. This affected users with multiple badges, as only one of their badges would be properly synchronized to the system, rendering their other badges non-functional for authentication purposes. ‌ ‌ **Resolution:** The issue will no longer occur for SaaS customers now that the mitigation efforts are completed as of September 5th 2025 at 9AM MDT. A small patch will be added September 5th, 2025, to ensure Virtual Appliance customers will not be affected in the next Virtual Appliance release, scheduled for November 3, 2025. ‌ ‌ **Mitigation:** A script to sync the badges for affected customers has resolved the issue for affected customers. Affected customers with the majority of PINs and badges were engaged to confirm resolution. The team expanded the effort to check and correct badge data for all other customers across all regions. ‌ ‌ **Conclusion:** This incident highlighted a gap in our synchronization process for users with multiple badges. The immediate fixes will resolve the issue for our SaaS customers, with preventative measures being implemented for Virtual Appliance customers prior to their next release cycle. Moving forward, we will implement more comprehensive testing for badge authentication scenarios, with a particular focus on edge cases, such as multiple badge assignments per user.

Read the full incident report →

Started Jun 25, 2025, 10:17 PM UTC· resolved Jun 25, 2025, 11:47 PM UTC

1. investigating Jun 25, 2025, 10:17 PM UTC

   We are currently investigating this issue.

2. monitoring Jun 25, 2025, 10:53 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 25, 2025, 11:47 PM UTC

   This incident has been resolved.

4. postmortem Jun 27, 2025, 07:28 PM UTC

   Wednesday, June 25, 2025 ‌ **SaaS instances experiencing intermittent 400/500 errors** ‌ ‌ **Issue Summary:** On 25 June 2025 between 9:38 PM UTC and 10:33 PM UTC some users with instances hosted in the US region experienced intermittent issues accessing the platform. ‌ **Root Cause:** A database node restarted unexpectedly and experienced high CPU and connection utilization once it came back online. ‌ **Resolution:** Data teams were able to identify the source of the high utilization and took action to resolve. ‌ **Mitigation:** To address high system utilization following unexpected database restarts, the Data team will promptly investigate and identify root causes, taking immediate corrective action to restore normal operations. Additionally, the team will validate all relevant system data and confirm the health of the database to ensure data integrity and system stability. ‌ **Conclusion:** We acknowledge this had an impact on some customers, we thank you for your patience as we resolved this service disruption.

Read the full incident report →

Started May 21, 2025, 08:47 PM UTC· resolved May 25, 2025, 03:55 AM UTC

1. investigating May 21, 2025, 08:47 PM UTC

   We are currently investigating this issue.

2. investigating May 22, 2025, 12:28 PM UTC

   We are continuing to investigate this issue.

3. identified May 23, 2025, 01:16 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 23, 2025, 02:10 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 25, 2025, 03:55 AM UTC

   This incident has been resolved.

6. postmortem May 27, 2025, 07:55 PM UTC

   **Issue Summary:** On May 21, 2025, a critical issue was identified where IdP groups and users were not syncing in Quota Management \(QM\). When selecting an IdP group in QM, the expected user list was returned empty, displaying no users to the end-user. Initially considered an instance-level database issue, it was immediately established as a Critical customer bug. After determining the affected regions, it was elevated to Blocking status on May 21, 2025, at 2:43 PM MT. ‌ **Root Cause:** The issue originated from a change implemented in Production on May 2, 2025, which affected the QM endpoint. The endpoint received an incorrect parameter, causing the failure to retrieve and display user lists. ‌ **Resolution:** Vasion’s Engineering teams resolved the issue by adjusting the calling procedures to pass the correct UUIDs instead of INTs. The fix was validated on May 22, 2025. It was then deployed to Production and validated on May 23, 2025. ‌ **Mitigation:** To prevent similar issues, QM testing will be included in future changes with Identity, given QM’s dependency on Identity. Each dependent service owner, including QM, will create and provide API tests for the Identity team to run whenever changes are made to Identity endpoint inputs or outputs. ‌ **Conclusion:** The issue was successfully resolved by correcting the parameter type passed to the endpoint, with the fix validated in Production. To enhance system reliability, integrating QM-specific API tests into the Vasion’s Engineering teams change process will help detect potential issues early, ensuring smoother operation of dependent services.

Read the full incident report →

Started May 14, 2025, 08:26 PM UTC· resolved May 15, 2025, 12:08 PM UTC

1. investigating May 14, 2025, 08:26 PM UTC

   We are currently investigating this issue.

2. investigating May 14, 2025, 10:56 PM UTC

   We are continuing to investigate this issue.

3. identified May 15, 2025, 01:47 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring May 15, 2025, 02:27 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 15, 2025, 12:08 PM UTC

   This incident has been resolved.

6. postmortem May 16, 2025, 04:45 PM UTC

   **IdP groups stopped working for printer deployment/portal security** ‌ ‌ **Issue Summary:** On 14 May 2025, at approximately 1:30 PM \(MDT\), the Vasion Print platform experienced a service disruption for printer deployment when using an Identity Provider \(IdP\) group. ‌ **Root Cause:** A data query was updated to pull data from a mirrored dataset. The mirrored dataset had not fully synced all of the original data. ‌ **Resolution:** All customers reporting the issue have been addressed by Vasion’s software and data engineers working to identify the missing data and running a sync of the original data into the proper mirrored dataset. ‌ **Mitigation:** Updates have been made to the mirroring process to ensure all data is properly synced in the future. **‌** **Conclusion:** No data has been lost during this incident. The underlying issue was due to data not fully syncing between two datasets. Each tenant for which the occurrence had been reported has been resolved. The underlying process has been updated to prevent further recurrence.

Read the full incident report →

Started Apr 16, 2025, 07:30 PM UTC· resolved Apr 16, 2025, 07:30 PM UTC

1. resolved May 14, 2025, 09:16 PM UTC

   Zoom reported outage affecting Vasion phone systems

Read the full incident report →

Started Apr 14, 2025, 02:00 PM UTC· resolved Apr 14, 2025, 04:30 PM UTC

1. resolved May 14, 2025, 09:15 PM UTC

   Intermittent issues signing in using SSO

2. postmortem May 14, 2025, 09:15 PM UTC

   **Intermittent issues signing in using SSO** ‌ ‌ **Issue Summary:** On 14 April 2025 between 2:00 PM UTC and 4:30 PM UTC, customers in our EMEA region experienced intermittent issues signing in using SSO: page would load blank, receive a 404 Bad Request response… ‌ **Root Cause:** A feature was enabled designed to add additional details for printing reports \(e.g. department, manager\). Enabling this feature added unintentional latency to these requests. ‌ **Resolution:** The underlying database query was identified, optimized, and tested, ensuring the additional latency was no longer being experienced. ‌ **Mitigation:** During the incident, infrastructure was increased to reduce latency while the data team analyzed the queries. **‌** **Conclusion:** We acknowledge the impact this had on customers within the EMEA region. We are committed to improving our processes to prevent future recurrence. Thank you for your patience as we resolved this service disruption.

Read the full incident report →

Started Apr 10, 2025, 10:30 PM UTC· resolved Apr 11, 2025, 01:23 PM UTC

1. resolved May 14, 2025, 09:14 PM UTC

   Unable to release print jobs when using OSRP

2. postmortem May 14, 2025, 09:14 PM UTC

   **Unable to release print jobs when using Offline Secure Release Printing \(OSRP\)** ‌ **Introduction:** On April 11, 2025, a subset of customers in the EMEA region encountered difficulties releasing held print jobs through the Control Panel Application \(CPA\) when the workstation client was offline. The issue was reported at 12:30am CEST, alerting our Product Support, DevOps, and Engineering teams. ‌ **Issue Summary:** During the incident, affected users were unable to release print jobs from the CPA interface. Other release mechanisms, such as the Release and Admin Portals and direct printing, continued to function as expected. The issue was specific to customers utilizing OSRP in conjunction with the CPA, and did not impact broader system functionality. ‌ **Root Cause:** A function essential for CPA-based job releases was absent in the latest deployment of a backend service. This issue manifested when testing combined CPA and OSRP specific workflows. ‌ **Resolution:** DevOps identified that a service deployment led to unexpected behavior impacting the CPA’s ability to release jobs. Reverting to the previous stable version of the service quickly restored functionality for affected customers and the incident was resolved as of 3:23pm CEST. ‌ **Mitigation:** * Reverting the service to its prior version resolved the issue and restored expected operations. * Subsequent monitoring confirmed that CPA job releases resumed successfully. ‌ **Conclusion:** We acknowledge the disruption caused to customers that rely on CPA for job releases. The rollback restored functionality quickly, and follow-up actions are being taken to improve coverage and testing for CPA-specific workflows. We are committed to preventing similar issues in the future and ensuring a reliable and consistent printing experience for all customers.

Read the full incident report →

Started Apr 01, 2025, 01:30 AM UTC· resolved Apr 01, 2025, 01:30 PM UTC

1. resolved May 14, 2025, 09:10 PM UTC

   SaaS instances experiencing intermittent 404 errors

2. postmortem May 14, 2025, 09:11 PM UTC

   **Issue Summary:** Between 11:20am and 1:30pm UTC, some instances in our Sydney region began experiencing intermittent 404 errors. During a routine scaling of our servers, new servers were not able to complete initialization leading to a subset of hosts becoming overwhelmed. As a result, a degraded experience for customers manifested through intermittent 404 errors from a portion of systems. **Resolution:** Our monitoring notified our team members who immediately began investigating the issue. Traffic was redirected from the affected systems to reduce the impact while the cause was further investigated. A reversion to a known working configuration was deployed while the identified cause was corrected. **Root Cause:** A recent change to an initialization script which is utilized during scaling events caused a significant delay during server startup which did not complete due to system load during the morning hours. **Mitigation:** After an audit of the adjustment, it was identified that the adjustment was safe to revert while the changes undergo further testing. We are performing additional testing on the proposed changes to ensure we will not experience delays during future scaling events. **Conclusion:** We acknowledge the impact this had on customers within the Sydney region. We are committed to improving our infrastructure scaling processes to prevent future recurrence. Thank you for your patience as we resolved this service disruption.

Read the full incident report →

Started Mar 06, 2025, 05:02 AM UTC· resolved Mar 06, 2025, 06:02 PM UTC

1. resolved May 14, 2025, 09:09 PM UTC

   Unable to login to customer instances from the MSP portal

2. postmortem May 14, 2025, 09:09 PM UTC

   **MSP portal not showing login option for instances** ‌ **Issue Summary:** On 06 Mar 2025 05:02UTC, a change to MSP Portal to improve error handling was tested and pushed into Production. Users were still able to log into the MSP Portal and Print Admin portals, but, as a result of this change, the functionality in MSP Portal that allows direct access to the Print Admin console was inadvertently disabled. This degradation required MSP Admins to directly log into Print Admin consoles instead of using the direct access button in MSP Portal. The Admin console direct access capability is heavily used by MSP Admins of Vasion products and led to many users being unable to access Print Admin consoles in the usual way. ‌ A fix to restore the direct access button for the Print Admin console in MSP Portal was pushed to Production at 2025-03-06 18:02UTC. **‌** **Root Cause:** The initial change to correct the undesired error condition had the side effect of hiding the user interface elements that allowed direct access to Print Admin console. Manual testing failed to discover the missing UI elements and led to the regression in capability. ‌ **Resolution:** Update the MSP Portal code to ensure the direct access capability was available for customers to use. In addition, more automated tests were added to catch future regression of this capability before these issues reach Production. **‌** **Conclusion:** To prevent further degradations of MSP Portal functionality, more automated tests were added to testing suites to confirm that the direct access button is present and working properly before allowing new changes in MSP Portal to reach Production.

Read the full incident report →

Started Feb 14, 2025, 04:00 PM UTC· resolved Feb 14, 2025, 04:00 PM UTC

1. resolved May 14, 2025, 09:06 PM UTC

   Intermittent issue with uploading drivers: Error occurred creating the driver package

2. postmortem May 14, 2025, 09:07 PM UTC

   \*\*Issue Summary:\*\*On February 14, 2025, users reported intermittent issues when attempting to upload driver packages. The error message displayed was: **"Error occurred creating the driver package."** ‌ \*\*Root Cause:\*\*The issue was identified as a missing query or function in the database, leading to a failed query or incomplete processing during driver package creation. Due to the intermittent nature of the problem, it was not immediately detected during manual testing. ‌ \*\*Resolution:\*\*We resolved the issue by reverting to a previous version of the code that included the necessary query/function. Additionally, we have incorporated enhanced testing into the build process to validate driver package creation, ensuring all required queries and functions are present in future updates. ‌ \*\*Conclusion:\*\*We acknowledge the impact this issue had on some customers' ability to use the product as expected. We remain committed to continuous improvement and are proactively working to minimize service disruptions moving forward.

Read the full incident report →