Hypercare Outage History

**What Happened?** At approximately 11:15 am EST on Monday, March 2, 2026, core services in Canada experienced a system-wide downtime. The incident was caused by database connection exhaustion. The team suspects this was caused by a background process responsible for resetting user statuses \(transitioning Hypercare users from “Unavailable” or “Busy” statuses back to “Available” after a set expiry date and time\) generated a high volume of sustained, long-lived connections. The database has a fixed limit on permissible concurrent connections and these hanging connections saturated the pool, preventing any new requests from being processed. ‌ **Impact** All Hypercare services were inaccessible for Canadian users from approximately 11:15 am EST until 12:22 pm EST on Monday, March 2, 2026. ‌ **Resolution and Next Steps** The Engineering team restored services by manually terminating stalled connections and increased the capacity on the database pool. The team disabled the automated status reset feature to stabilize the environment and allow core services to resume normal operation. The automated status reset feature has been running intermittently in a controlled environment to ensure users status’ are reset appropriately.. ‌ To reduce the chances of a recurrence and improve our response time, the following actions are being taken: * Enhanced Monitoring: We are implementing additional early detection alerts for database connection utilization. This will allow us to intervene before the limit is reached. * Infrastructure Updates: We are increasing the number of permissible connections to the database to facilitate faster recovery during database restarts. * Rapid Recovery Protocol: While we finalize the permanent root cause fix, we have implemented a rapid recovery protocol, which will allow the team to instantly clear hanging connections, reducing potential recovery time from several minutes to under 60 seconds. ‌ We apologize for the disruption caused by today’s unplanned downtime. We thank everyone for their patience and continued support.

**What Happened?** Between 00:00 and 02:00 EST on February 14, 2026, push notifications for mobile app users \(iOS and Android\) in Canada were not being delivered. The incident was triggered by a non-routine database configuration change that was successfully completed at midnight on February 14 in preparation for an upcoming deployment. However, several backend services that relied on the affected databases were caching connections and related configuration data, which resulted in a conflict between the services and the database. While messages were successfully sent within the app \(i.e. active conversations continued uninterrupted at that time\), push notification alerts failed to deliver, which may prevent a user from realizing they had new incoming messages. The issue was reported at 01:44 EST and resolved by 02:00 EST through a service restart. **Impact** Users in Canada experienced interruption of real-time alerts between 00:00 and 02:00 EST. The scope of the impact included: * Hypercare messaging notifications: Interruptions in push notifications for secure messages. * Virtual paging: Interruptions in virtual paging services and real-time alerts. **Resolution and Next Steps** The immediate resolution was achieved by restarting the services. To prevent recurrence and improve detection, the following actions have been implemented: * SOP Updates: Documentation has been updated to prioritize service-restart protocols specifically following any database failover. * Enhanced Post-Deployment Testing: Updated deployment checklists to include specific push notification testing for iOS and Android. * Granular Monitoring: Adjusted internal alerts sensitivity. Alerts are now configured as recurring every 10 minutes to ensure higher visibility and faster response. * Redundant Internal Alerting: Implemented secondary notification channels to ensure rapid response if the primary Hypercare notification pathway is degraded.