BarTender Cloud experienced a minor incident on May 21, 2025, lasting 1h 40m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved May 21, 2025, 08:15 PM UTC
At approximately 06:22 PM PDT, we experienced an issue with one of our internal services that temporarily impacted the performance of the Print API. While the Print API remained available, some users may have experienced slower responses or delays in processing print jobs. Our team quickly identified the issue and worked to restore normal service. The problem has since been resolved by 09:05 PM PDT, and all systems are operating normally. We apologize for any inconvenience this may have caused and appreciate your understanding.
- postmortem May 22, 2025, 12:04 AM UTC
**AM1- BTC API Printing Degradation** **Date:** May 20, 2025 **Time:** 18:50 PDT to 20:30 PDT **Status:** RESOLVED **Summary** On May 20, 2025, from 6:50 PM to 8:30 PM PDT, the BTC Print API in the AM1 region experienced a temporary outage that impacted printing operations for some users. The issue was caused by a failure in one of the core services responsible for processing API requests. The system has since recovered, and normal functionality has been restored. **What Happened** A failure occurred in the inbound gateway service, which handles incoming API calls related to printing. This failure affected the service’s ability to process requests, leading to a partial outage of the API printing functionality. The issue was initially silent, there were no immediate failure alerts, resulting in a delay before detection and resolution. **Resolution** Once the issue was identified, our team quickly redeployed all instances of the affected service to restore stability and resume operations. We implemented fixes in the service code to enhance the reliability and responsiveness of the health and readiness endpoints, ensuring early detection and reporting of failures. Additionally, a new metric has been implemented to detect silent failures automatically, enabling proactive intervention and minimizing service downtime. **Next Steps** To prevent recurrence and improve detection, we have taken the following actions: * Fixed the gateway service’s health check mechanisms to better detect failures early. * Automated the recovery process for the inbound gateway service to reduce recovery time in future incidents. * Continued investment in monitoring enhancements to improve visibility across all critical services. We sincerely apologize for the inconvenience and appreciate your patience while we addressed the issue.