"Bad Gateway" error when accessing SRS instances

Update timeline

1. investigating Dec 17, 2025, 03:27 PM UTC

   We are currently investigating this issue.

2. identified Dec 17, 2025, 04:28 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Dec 17, 2025, 05:25 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Dec 17, 2025, 06:14 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Dec 17, 2025, 07:15 PM UTC

   This incident has been resolved.

6. postmortem Dec 19, 2025, 07:10 PM UTC

   **US - Bad Network Gateway interruption** ‌ **Issue Summary:** On December 17th, from 8:00 AM MST to 12:15 PM MST, all services requiring access to our hosted environment in the US region \([printercloud.com](http://printercloud.com)\) were unavailable, returning a 500 Bad Gateway error. Local services not requiring connections to our systems remained functional. The outage was caused by exhausted system resources due to a system package upgrade, preventing our systems from processing incoming requests. ‌ **Root Cause:** During routine system patching, the containerd package was updated, introducing breaking changes to resource management that caused cascading service failures across production environments. Services exhausted available system resources and could no longer accept network connections, causing the platform to become unstable. ‌ **Resolution:** While troubleshooting, the operations team applied a configuration override to restore appropriate system resource limits. Once the offending update was identified, the operations team rolled back the system-level deployment package. ‌ **Mitigation:** We are implementing staged deployment processes for system updates with mandatory testing in non-production environments, and expanding monitoring to alert on system configuration changes before potential impact to production services. As a next step, our Operations team will decouple OS-level security patches and package updates from Schedule Release SaaS application releases. This will allow us to: ‌ * Maintain our security posture with timely OS patches * Reduce the complexity and risk profile of application deployments * Improve our ability to identify root causes quickly if issues occur * Enable faster rollbacks if problems arise **‌** **Conclusion:** We acknowledge the impact this had on customers in the US region. We are committed to improving our processes to prevent future recurrence. Thank you for your understanding as our teams worked to resolve the issues stemming from this container package upgrade.