Hypercare experienced a critical incident on February 20, 2025 affecting Notifications and Real-Time Syncing (Canadian Region), lasting 15h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 20, 2025, 12:01 AM UTC
We are currently investigating reports of Hypercare message notifications experiencing delays in delivery. Our team is actively investigating the issue. This is only affecting users in Canada.
- monitoring Feb 20, 2025, 12:08 AM UTC
Delivery of notifications have resumed normal operations. Our team is continuing the investigation and conducting additional testing.
- monitoring Feb 20, 2025, 12:28 AM UTC
We are continuing to monitor for any further issues.
- resolved Feb 20, 2025, 03:11 PM UTC
This incident has been resolved.
- postmortem Feb 20, 2025, 03:12 PM UTC
**What Happened?** At 18:50 EST, our monitoring system flagged a potential issue with the notification service in Hypercare. An immediate investigation was launched by the team. Testing concluded the issue was isolated to users based in Canada. **Impact** The notification service interruption affected several critical functionalities within Hypercare, including: delivery of Hypercare message notifications within the platform, outgoing pages and SMS sent through Hypercare and integrations with some third-party services, such as Everbridge. The investigation indicated that messages sent between 18:34 and 19:05 EST experienced prolonged delays of up to 20 minutes in notification delivery. **Resolution and Next Steps** The issue was resolved at 19:05 EST. Our team is continuing to monitor the notification service and conducting further investigation on the root cause.