Hypercare incident

[US and Canada Regions] Partial Service Interruption Due to Third-Party Outage

Major Resolved View vendor source →

Hypercare experienced a major incident on June 12, 2025 affecting Virtual Pager (U.S. Region) and Notifications and Real-Time Syncing (Canadian Region) and 1 more component, lasting 1h 16m. The incident has been resolved; the full update timeline is below.

Started
Jun 12, 2025, 07:26 PM UTC
Resolved
Jun 12, 2025, 08:42 PM UTC
Duration
1h 16m
Detected by Pingoru
Jun 12, 2025, 07:26 PM UTC

Affected components

Virtual Pager (U.S. Region)Notifications and Real-Time Syncing (Canadian Region)Notifications and Real-Time Syncing (U.S. Region)Code Teams (Canadian Region)Code Teams (U.S. Region)Administration and Scheduling (Canadian Region)Administration and Scheduling (U.S. Region)API & Integrations (Canadian Region)API & Integrations (U.S. Region)Virtual Pager (Canadian Region)

Update timeline

  1. investigating Jun 12, 2025, 07:26 PM UTC

    We are currently experiencing a partial service disruption caused by an ongoing outage with Doppler, one of our third-party service providers. As a result, the following Hypercare features are impacted: - Escalation notifications related to code team activations (note: code team activations remain fully operational) - Schedule change email notifications - Schedule and Equity report downloads - Everbridge file generation We are actively monitoring the situation and will provide updates as Doppler resolves the issue. You can view the status of the Doppler incident here: https://www.dopplerstatus.com/incidents/8jtzf7nrl9tv

  2. identified Jun 12, 2025, 07:44 PM UTC

    The engineering team is manually updating the secrets to restore the services. Everbridge integration has been restored.

  3. identified Jun 12, 2025, 07:51 PM UTC

    In addition to the previously reported Doppler outage, we have now identified a separate issue affecting virtual paging. This is due to a service disruption with Twilio, a third-party vendor. As a result, virtual paging is currently unavailable and may affect message delivery for some users. You can view the Twilio status here: https://status.twilio.com We continue to monitor both the Doppler and Twilio incidents and will provide updates as services are restored. Thank you for your continued patience.

  4. identified Jun 12, 2025, 08:09 PM UTC

    Automated code escalation has been manually restored. All code team functions are now fully operational.

  5. monitoring Jun 12, 2025, 08:29 PM UTC

    The third-party services have been recovered. All Hypercare services are operational. We're currently monitoring the services.

  6. resolved Jun 12, 2025, 08:42 PM UTC

    The outage with the third party vendor has been resolved.