[CA Region only] Push Notification Delivery Issues

Update timeline

1. resolved Feb 23, 2026, 05:28 PM UTC

   Between 00:00 and 02:00 AM EST on February 14, users in our Canadian region experienced an interruption with Hypercare’s push notification services. During this window, messages were successfully delivered to the intended recipient, however, real-time alerts (push notifications) were not being sent to iOS and Android devices. Hypercare recognized the issue at 01:44 AM and the root cause was identified at 1:58 AM. A technical fix was applied and notification services were fully restored by 02:00 AM.

2. postmortem Feb 23, 2026, 05:28 PM UTC

   **What Happened?** Between 00:00 and 02:00 EST on February 14, 2026, push notifications for mobile app users \(iOS and Android\) in Canada were not being delivered. The incident was triggered by a non-routine database configuration change that was successfully completed at midnight on February 14 in preparation for an upcoming deployment. However, several backend services that relied on the affected databases were caching connections and related configuration data, which resulted in a conflict between the services and the database. While messages were successfully sent within the app \(i.e. active conversations continued uninterrupted at that time\), push notification alerts failed to deliver, which may prevent a user from realizing they had new incoming messages. The issue was reported at 01:44 EST and resolved by 02:00 EST through a service restart. **Impact** Users in Canada experienced interruption of real-time alerts between 00:00 and 02:00 EST. The scope of the impact included: * Hypercare messaging notifications: Interruptions in push notifications for secure messages. * Virtual paging: Interruptions in virtual paging services and real-time alerts. **Resolution and Next Steps** The immediate resolution was achieved by restarting the services. To prevent recurrence and improve detection, the following actions have been implemented: * SOP Updates: Documentation has been updated to prioritize service-restart protocols specifically following any database failover. * Enhanced Post-Deployment Testing: Updated deployment checklists to include specific push notification testing for iOS and Android. * Granular Monitoring: Adjusted internal alerts sensitivity. Alerts are now configured as recurring every 10 minutes to ensure higher visibility and faster response. * Redundant Internal Alerting: Implemented secondary notification channels to ensure rapid response if the primary Hypercare notification pathway is degraded.