BarTender Cloud experienced a major incident on May 20, 2025 affecting AM1 - Print Engine Services, lasting 2h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 20, 2025, 01:38 PM UTC
We are currently experiencing errors when accessing the Print Portal which is causing a partial outage for some customers. Printing via the API remains operational. Our team is actively investigating the issue and working to restore full functionality.
- investigating May 20, 2025, 02:40 PM UTC
We are continuing to investigate this issue.
- investigating May 20, 2025, 03:42 PM UTC
We are continuing to investigate this issue.
- resolved May 20, 2025, 04:28 PM UTC
This incident has been resolved.
- postmortem May 21, 2025, 11:48 PM UTC
**AM1 – BTC Print Portal Access Issues** **Dates:** May 16, 19, and 20, 2025 **Times:** * May 16: 11:27 AM – 12:16 PM PDT * May 19: 07:36 AM – 08:18 AM PDT * May 19: 11:10 AM – 11:27 AM PDT * May 20: 06:30 AM – 09:28 AM PDT **Status:** Resolved **Summary** At specific times on May 16, 19, and 20, some users experienced difficulty accessing the BTC Print Portal in the AM1 region. These intermittent issues were due to temporary service degradation during peak traffic periods. All services have since been restored and are operating normally. **What Happened** The access issues were caused by a limitation in how the Print Portal server handled increased activity. During each of the incidents, one server instance became resource-starved and was unable to handle incoming requests, resulting in degraded performance and intermittent unavailability for users accessing the portal. **Resolution** To mitigate the impact, our team restarted the affected services during each event to restore access. After identifying the root cause, we applied a configuration fix to address the scaling issue. Additional supporting services are also being introduced to help offload internal calls and improve performance under load. **Next Steps** To prevent recurrence and improve reliability, we have: * Implemented enhanced monitoring and alerting on the load balancer that distributes traffic across portal servers. * Set proactive capacity thresholds to detect early signs of performance issues. * Continued work to introduce supporting services that reduce pressure on the main portal components. We appreciate your patience and understanding while we worked to resolve this issue.