Hypercare Outage History

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   Virtual paging services were restored to normal operations at 13:59 ET Impacted: Virtual Pager - US Region

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident has been resolved. Impacted: Login and Single Sign On - US Region, Messaging - US Region, Notifications and Real-Time Syncing - US Region, +7 more

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident has been resolved. Impacted: Login and Single Sign On - US Region, Messaging - US Region, Notifications and Real-Time Syncing - US Region, +7 more

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident has been resolved. Impacted: Login and Single Sign On - US Region, Messaging - US Region, Notifications and Real-Time Syncing - US Region, +7 more

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident is fully resolved. Our team will continue to monitor to ensure this incident does not occur again. Impacted: Notifications and Real-Time Syncing - US Region

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident has been resolved. Impacted: Login and Single Sign On - US Region, Messaging - US Region, Notifications and Real-Time Syncing - US Region, +7 more

Read the full incident report →

Started Jul 08, 2026, 08:07 PM UTC· resolved Jul 08, 2026, 08:07 PM UTC

1. resolved Jul 08, 2026, 08:07 PM UTC

   This incident has been resolved. Impacted: Self-serve Scheduling - US Region

Read the full incident report →

Started Jul 05, 2026, 03:21 AM UTC· resolved Jul 05, 2026, 11:16 AM UTC

1. investigating Jul 05, 2026, 03:21 AM UTC

   We're observing intermittent interruptions to Hypercare services. We're investigating the issue.

2. identified Jul 05, 2026, 03:31 AM UTC

   The issue has been identified to be related to a cloud service maintenance.

3. identified Jul 05, 2026, 03:47 AM UTC

   The engineers have restarted the service. Services are back to operational. We're monitoring the cloud service maintenance progress.

4. monitoring Jul 05, 2026, 04:12 AM UTC

   Services are back to operational since 11:30pm. We're monitoring the cloud service maintenance progress.

5. resolved Jul 05, 2026, 11:16 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 01, 2026, 01:43 PM UTC· resolved Jun 01, 2026, 04:51 PM UTC

1. monitoring Jun 01, 2026, 01:43 PM UTC

   We noticed an intermittent slowdown for Hypercare services in the Canada region at 9:11am. The issue has been identified and implemented a fix. All services are operational as at 9:28am and we're closely monitoring.

2. identified Jun 01, 2026, 02:03 PM UTC

   We're seeing slowdown to some Hypercare services. We're looking into the issue.

3. monitoring Jun 01, 2026, 02:20 PM UTC

   All services are operational. We're monitoring the services closely.

4. resolved Jun 01, 2026, 04:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 29, 2026, 03:27 PM UTC· resolved May 29, 2026, 04:41 PM UTC

1. investigating May 29, 2026, 03:27 PM UTC

   We identified an issue with file attachment uploads on the Hypercare platform in the Canada region. We're currently investigating the issue.

2. identified May 29, 2026, 03:39 PM UTC

   We have identified the issue and working on a fix.

3. monitoring May 29, 2026, 03:49 PM UTC

   A fix has been implemented and file upload is operational.

4. monitoring May 29, 2026, 04:17 PM UTC

   The incident is resolved.

5. resolved May 29, 2026, 04:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 03, 2026, 01:36 AM UTC· resolved May 03, 2026, 03:27 AM UTC

1. identified May 03, 2026, 01:36 AM UTC

   A service interruption has been identified impacting real time notifications to the Hypercare platform. This issue is caused by maintenance work from Hypercare's cloud services provider. An update is expected in the next 10 minutes.

2. monitoring May 03, 2026, 01:41 AM UTC

   Real time notifications are restored and we are continuing to monitor the issue.

3. identified May 03, 2026, 02:21 AM UTC

   Partial interruption to services are being experienced due to maintenance work on Hypercare's cloud services provider. Intermittent interruptions may be experienced in the next 5-10 minutes.

4. monitoring May 03, 2026, 02:30 AM UTC

   All core services are now operational and we are continuing to monitor.

5. resolved May 03, 2026, 03:27 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 29, 2026, 10:01 PM UTC· resolved Mar 30, 2026, 06:30 PM UTC

1. investigating Mar 29, 2026, 09:44 PM UTC

   We are currently experiencing downtime to all Hypercare services. The team is investigating the issue.

2. monitoring Mar 29, 2026, 10:01 PM UTC

   All services are operational. We have identified the issue and we're closely monitoring.

3. monitoring Mar 29, 2026, 10:25 PM UTC

   The schedule tab on web client and admin portal is not loading. Our team is investigating.

4. monitoring Mar 29, 2026, 10:36 PM UTC

   We observed increased connections to our database which is causing issues. Users might experience intermittent errors or a longer loading time when using the application. We're monitoring the issue.

5. monitoring Mar 30, 2026, 12:24 AM UTC

   All services are operational. We're monitoring the issue.

6. monitoring Mar 30, 2026, 12:37 AM UTC

   We are noticing an inconsistent delivery for real time messages. Push notifications are not impacted. We're restarting the services.

7. monitoring Mar 30, 2026, 01:11 AM UTC

   Services are back to operational. We will share a post-mortem shortly.

8. resolved Mar 30, 2026, 06:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 04, 2026, 05:57 PM UTC· resolved Mar 06, 2026, 04:30 AM UTC

1. investigating Mar 04, 2026, 05:57 PM UTC

   We are currently investigating an unplanned downtime of all core services for users in Canada.

2. monitoring Mar 04, 2026, 06:44 PM UTC

   A fix has been implemented and core services have returned. We are continuing to monitor the incident.

3. identified Mar 04, 2026, 07:14 PM UTC

   We are observing intermittent latency in push notifications and real-time syncing of messages on web client. We're monitoring the system performance closely.

4. monitoring Mar 04, 2026, 09:06 PM UTC

   A fix has been implemented. Real time notification and syncing are operational. We are monitoring the performance.

5. resolved Mar 06, 2026, 04:30 AM UTC

   Core services have resumed and remained stable for the past 24 hours. The team is continuing to monitor push notification and web socket events for real-time syncing. The team has continued to disable the automated status reset feature to maintain a stable environment. The automated status reset feature has been running intermittently in a controlled environment to ensure users status’ are reset appropriately.

Read the full incident report →

Started Mar 04, 2026, 08:57 AM UTC· resolved Mar 04, 2026, 09:57 AM UTC

1. investigating Mar 04, 2026, 08:57 AM UTC

   We are currently investigating an issue with push notifications impacting users in Canada. Push notifications are being reported as delivering intermittently to users of the Hypercare mobile app and web platform.

2. investigating Mar 04, 2026, 09:24 AM UTC

   We are continuing to investigate an issue with push notifications impacting users in Canada. Users of the Hypercare mobile and web platform are receiving intermittent push notifications. Messages to physical pagers have also been identified as being impacted by this issue.

3. monitoring Mar 04, 2026, 09:30 AM UTC

   The issue has been identified and a fix has been implemented.

4. resolved Mar 04, 2026, 09:57 AM UTC

   This incident has been resolved. A postmortem will be posted shortly.

Read the full incident report →

Started Mar 03, 2026, 06:34 AM UTC· resolved Mar 30, 2026, 08:26 PM UTC

1. identified Mar 03, 2026, 06:34 AM UTC

   We have temporarily disabled the Directory Synchronization feature (automatic user provisioning and deprovisioning) in both Canada and US regions, as it has been identified that configuration adjustments are needed to ensure the feature operates in full capacity within our new cloud environment. A fix is underway and a full sync will be ran against applicable organizations when the feature is re-enabled. For users that need to be on an on-call schedule prior to the fix being released, organization administrators are advised to create accounts manually for users via the Admin Portal (admin.hypercare.com). Alternatively, you may contact Hypercare Support ([email protected] or 1(877) 379-3533) for advice on next steps. Users can continue to log into their organizations with Single Sign-On (SSO), if enabled. SSO is not impacted by this incident.

2. resolved Mar 30, 2026, 08:26 PM UTC

   Directory Synchronization has been enabled. Following the deployment of a fix on March 26, we have completed the following recovery steps: - Full Synchronization: All Active Directory records were resynced between March 26-28 to ensure data accuracy. - Real-time Updates: Webhooks are now active and capturing directory changes (adds, moves, and changes) as they occur. - Final Phase: We are currently finalizing the enablement for a small subset of organizations with specific shared-account configurations. Organization administrators have been sent a detailed briefing regarding their specific sync status. If you have immediate questions, please contact [email protected].

Read the full incident report →

Started Mar 02, 2026, 04:25 PM UTC· resolved Mar 02, 2026, 06:35 PM UTC

1. investigating Mar 02, 2026, 04:25 PM UTC

   We are currently investigating an unplanned downtime of all core services.

2. identified Mar 02, 2026, 04:46 PM UTC

   The issue has been identified and we are working on a fix.

3. monitoring Mar 02, 2026, 05:23 PM UTC

   A fix has been released and all core services have been restored. We're continuing to monitor the incident.

4. resolved Mar 02, 2026, 06:35 PM UTC

   This incident has been resolved. A post mortem will be shared shortly.

5. postmortem Mar 03, 2026, 06:17 AM UTC

   **What Happened?** At approximately 11:15 am EST on Monday, March 2, 2026, core services in Canada experienced a system-wide downtime. The incident was caused by database connection exhaustion. The team suspects this was caused by a background process responsible for resetting user statuses \(transitioning Hypercare users from “Unavailable” or “Busy” statuses back to “Available” after a set expiry date and time\) generated a high volume of sustained, long-lived connections. The database has a fixed limit on permissible concurrent connections and these hanging connections saturated the pool, preventing any new requests from being processed. ‌ **Impact** All Hypercare services were inaccessible for Canadian users from approximately 11:15 am EST until 12:22 pm EST on Monday, March 2, 2026. ‌ **Resolution and Next Steps** The Engineering team restored services by manually terminating stalled connections and increased the capacity on the database pool. The team disabled the automated status reset feature to stabilize the environment and allow core services to resume normal operation. The automated status reset feature has been running intermittently in a controlled environment to ensure users status’ are reset appropriately.. ‌ To reduce the chances of a recurrence and improve our response time, the following actions are being taken: * Enhanced Monitoring: We are implementing additional early detection alerts for database connection utilization. This will allow us to intervene before the limit is reached. * Infrastructure Updates: We are increasing the number of permissible connections to the database to facilitate faster recovery during database restarts. * Rapid Recovery Protocol: While we finalize the permanent root cause fix, we have implemented a rapid recovery protocol, which will allow the team to instantly clear hanging connections, reducing potential recovery time from several minutes to under 60 seconds. ‌ We apologize for the disruption caused by today’s unplanned downtime. We thank everyone for their patience and continued support.

Read the full incident report →

Started Feb 25, 2026, 10:14 AM UTC· resolved Feb 26, 2026, 01:46 AM UTC

1. identified Feb 25, 2026, 10:14 AM UTC

   An issue has been identified regarding shifts not populating in Hypercare for departments that leverage a third-party schedule integration partner (MetricAid and Mesh AI), which is impacting users in Canada. If there are any changes that need to be made to the current on-call shift(s), schedule/department admins will need to make these updates directly in Hypercare's Admin Portal (admin.hypercare.com) to ensure the correct information appears in Hypercare. A fix is being worked on. Next update is expected for 12:00 PM or sooner. After the fix is deployed, schedule information will be re-synced to ensure data is accurate in Hypercare.

2. identified Feb 25, 2026, 06:34 PM UTC

   We are continuing to work on a fix for this issue. If there are any changes that need to be made to the current on-call shift(s), schedule/department admins will need to make these updates directly in Hypercare's Admin Portal (admin.hypercare.com) to ensure the correct information appears in Hypercare.

3. resolved Feb 26, 2026, 01:46 AM UTC

   This issue has been resolved.

Read the full incident report →

Started Feb 14, 2026, 05:00 AM UTC· resolved Feb 14, 2026, 05:00 AM UTC

1. resolved Feb 23, 2026, 05:28 PM UTC

   Between 00:00 and 02:00 AM EST on February 14, users in our Canadian region experienced an interruption with Hypercare’s push notification services. During this window, messages were successfully delivered to the intended recipient, however, real-time alerts (push notifications) were not being sent to iOS and Android devices. Hypercare recognized the issue at 01:44 AM and the root cause was identified at 1:58 AM. A technical fix was applied and notification services were fully restored by 02:00 AM.

2. postmortem Feb 23, 2026, 05:28 PM UTC

   **What Happened?** Between 00:00 and 02:00 EST on February 14, 2026, push notifications for mobile app users \(iOS and Android\) in Canada were not being delivered. The incident was triggered by a non-routine database configuration change that was successfully completed at midnight on February 14 in preparation for an upcoming deployment. However, several backend services that relied on the affected databases were caching connections and related configuration data, which resulted in a conflict between the services and the database. While messages were successfully sent within the app \(i.e. active conversations continued uninterrupted at that time\), push notification alerts failed to deliver, which may prevent a user from realizing they had new incoming messages. The issue was reported at 01:44 EST and resolved by 02:00 EST through a service restart. **Impact** Users in Canada experienced interruption of real-time alerts between 00:00 and 02:00 EST. The scope of the impact included: * Hypercare messaging notifications: Interruptions in push notifications for secure messages. * Virtual paging: Interruptions in virtual paging services and real-time alerts. **Resolution and Next Steps** The immediate resolution was achieved by restarting the services. To prevent recurrence and improve detection, the following actions have been implemented: * SOP Updates: Documentation has been updated to prioritize service-restart protocols specifically following any database failover. * Enhanced Post-Deployment Testing: Updated deployment checklists to include specific push notification testing for iOS and Android. * Granular Monitoring: Adjusted internal alerts sensitivity. Alerts are now configured as recurring every 10 minutes to ensure higher visibility and faster response. * Redundant Internal Alerting: Implemented secondary notification channels to ensure rapid response if the primary Hypercare notification pathway is degraded.

Read the full incident report →

Started Jan 11, 2026, 01:26 AM UTC· resolved Jan 11, 2026, 01:44 AM UTC

1. investigating Jan 11, 2026, 01:26 AM UTC

   We are currently investigating alerts that push notifications are not being delivered to mobile app users (iOS and Android). This is only impacting users in Canada. Our engineering team is looking into the root cause, and we will provide an update as soon as we have more information.

2. monitoring Jan 11, 2026, 01:34 AM UTC

   A fix has been implemented and push notifications are now being delivered. We will continue monitoring to ensure a full resolution.

3. monitoring Jan 11, 2026, 01:34 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Jan 11, 2026, 01:44 AM UTC

   This incident has been resolved. A post mortem will be posted shortly.

5. postmortem Jan 14, 2026, 02:14 PM UTC

   **What Happened?** At 20:10 EST, the Customer Support and Engineering teams were alerted to ongoing delays in push notifications specifically affecting users in Canada. The incident coincided with a non-routine manual backup of Canadian production databases, which was being performed in preparation for upcoming deployments. The manual backup process consumed significant system resources, creating a bottleneck that caused significant latency in our notification services. Once a potential correlation between the backup and the latency was identified, the team immediately halted all manual backup operations. Following the termination of these tasks, system performance stabilized, and services returned to normal operations by 20:35 EST. ‌ **Impact** Users located in Canada likely experienced significant latency in notification delivery across several Hypercare services between 19:00 and 20:30 EST. The scope of the impact included: * Hypercare messaging: Delays in receiving push notifications for Hypercare secure messages. * SMS services: Latency in SMS messages sent from the Hypercare platform. * Escalation phone calls: Delays in escalation phone calls triggered by code team activations. * Virtual paging: Interruptions in virtual paging services. ‌ **Resolution and Next Steps** The immediate resolution was achieved by stopping the manual database backup. As this was a non-routine event, existing automated monitors did not initially flag the resource exhaustion as a service-level threat. To prevent a recurrence and improve detection, the following actions have been implemented: * Enhanced monitoring: New, granular alerts have been configured to identify server-specific bottlenecks and resource spikes at the database level, providing an earlier warning system for manual interventions. * Alternative non-routine database backup procedures: We have identified and adopted an alternative methodology for manual production database backups that offloads the process from the primary live environment to avoid impacting real-time traffic.

Read the full incident report →

Started Oct 29, 2025, 09:00 AM UTC· resolved Nov 06, 2025, 07:40 PM UTC

1. identified Oct 29, 2025, 09:00 AM UTC

   As part of Hypercare’s recent upgrade to user account structures, the scheduled maintenance downtime has been extended beyond 05:00 EDT. All Hypercare services continue to be temporarily unavailable on our mobile applications (iOS and Android). These include, but are not limited to messaging, scheduling, on-call locating, and code activations. Our Web platforms (app.hypercare.com and admin.hypercare.com) are fully operational. We apologize for the inconvenience and are working on restoring all services.

2. monitoring Oct 29, 2025, 10:13 AM UTC

   All Hypercare services have resumed on all platforms. Users who are part of two or more Hypercare organizations are advised to verify they have remained logged in to all their desired Hypercare organizations, as these users may have been logged out during the maintenance period. Our team will continue to monitor services returning to operations.

3. monitoring Oct 29, 2025, 10:14 AM UTC

   Virtual paging services continue to be redirected to designated alternate numbers and we are working on resuming regular service.

4. investigating Oct 29, 2025, 10:22 AM UTC

   Escalations phone calls for code team activations is currently experiencing an issue. The team is investigating the root cause.

5. monitoring Oct 29, 2025, 10:25 AM UTC

   Escalations phone calls for code team activations has returned to operations.

6. monitoring Oct 29, 2025, 10:45 AM UTC

   Virtual Paging has returned to operations.

7. monitoring Oct 29, 2025, 07:33 PM UTC

   We are receiving support that some of the users are not able to view the list of chats from their account. We are currently investigating the cause of this issue and our engineering team is currently identifying a resolution.

8. identified Oct 30, 2025, 04:24 AM UTC

   We are aware of several issues following the recent deployment and our team is actively working to restore full service availability. The following functionalities are currently impacted: - (CA only) Some users are unable to load or view their chat list - (CA only) Some administrators are unable to access schedules - (CA only) Schedule and equity report downloads are temporarily unavailable - (CA only) External schedule integration is paused - (CA only) Backup SMS delivery is currently unavailable - (CA and US) Directory Synchronization is paused We will provide an update as soon as these services are restored. Thank you for your patience and understanding.

9. identified Oct 30, 2025, 03:47 PM UTC

   The following issue has been resolved: - (CA only) Some users are unable to load or view their chat list The following functionalities are currently impacted, and the team is actively working on: - (CA only) Some administrators are unable to access schedules - (CA only) Schedule and equity report downloads are temporarily unavailable - (CA only) External schedule integration is paused - (CA only) Backup SMS delivery is currently unavailable - (CA and US) Directory Synchronization is paused We will provide an update as soon as these services are restored. Thank you for your patience and understanding.

10. identified Oct 30, 2025, 04:26 PM UTC

    The following issue has been resolved: - (CA only) Some users are unable to load or view their chat list - NEW (CA only) Some administrators are unable to access schedules - NEW (CA only) Equity report downloads The following functionalities are currently impacted, and the team is actively working on: - (CA only) Schedule download is temporarily unavailable - (CA only) External schedule integration is paused - (CA only) Backup SMS delivery is currently unavailable - (CA and US) Directory Synchronization is paused We will provide an update as soon as these services are restored. Thank you for your patience and understanding.

11. identified Oct 30, 2025, 06:00 PM UTC

    The following issue has been resolved: - (CA only) Some users are unable to load or view their chat list - (CA only) Some administrators are unable to access schedules - (CA only) Equity report downloads - UPDATED (CA only) Schedule download is temporarily unavailable - UPDATED (CA only) Backup SMS delivery is currently unavailable The following functionalities are currently impacted, and the team is actively working on: - (CA only) External schedule integration is paused - (CA and US) Directory Synchronization is paused

12. identified Oct 31, 2025, 04:39 PM UTC

    The following issue has been resolved: - (CA only) Chat list loading issues - (CA only) Schedule access issues for administrators - (CA only) Equity report downloads - (CA only) Schedule download - (CA only) Backup SMS delivery - UPDATED (CA only) External schedule integration The following functionalities are currently impacted, and the team is actively working on: - (CA and US) Directory Synchronization is paused - NEW (CA and US) Some users may not receive push notifications since the migration. Recommended action: Users should remove and log out of all accounts from the device, then log in again.

13. identified Nov 01, 2025, 02:53 AM UTC

    The following issues have been resolved: - (CA and US) Nov 1 on-call shifts displayed an error when viewing the on-call schedule on Android. - (CA and US) Android app version 15.1.7 was released to resolve a notification bug impacting some US Region users. The following functionalities continue to remain impacted and the team is actively working on restoring full services: - (CA and US) Directory Synchronization is paused - (CA and US) Some users have reported not receiving push notifications to their iOS or Android app since the migration. For users experiencing this, ensure the mobile app is updated to the latest version (iOS app 18.4.2 and Android app 15.1.7), log out and remove all accounts from the device, close the app, and then log in again. If immediate assistance is needed, users can call 1(877) 379-3533. This issue has not been widely reported, so we continue to monitor reported cases. If users are receiving push notifications, no further action is required.

14. identified Nov 04, 2025, 01:43 AM UTC

    The following issues have been resolved: - (CA and US) Directory Synchronization has been turned back on - (CA and US) Shell accounts are now viewable on the On-Call page on iOS The following functionalities continue to remain impacted and the team is actively working on restoring full services: - (CA and US) Some users have reported not receiving push notifications to their iOS or Android app since the migration. For users experiencing this, ensure the mobile app is updated to the latest version, log out and remove all accounts from the device, close the app, and then log in again. If immediate assistance is needed, users are advised to call 1(877) 379-3533. This issue has not been widely reported, so we continue to monitor reported cases. If users are receiving push notifications, no further action is required.

15. resolved Nov 06, 2025, 07:40 PM UTC

    All critical Hypercare services have returned to normal operations. If users are experiencing any technical issues, call 1(877) 379-3533 for immediate assistance or email [email protected] for non-urgent inquiries.

Read the full incident report →

Started Oct 28, 2025, 10:20 AM UTC· resolved Oct 28, 2025, 03:50 PM UTC

1. investigating Oct 28, 2025, 10:20 AM UTC

   [CA Region, Web only] The web application in Canada region is having the below issues: - Login via single-sign-on. Logged in users are advised to stay logged in - Viewing the contact profile details of users. Searching users is impacted Mobile users are not impacted. The engineers are currently investigating the issues.

2. identified Oct 28, 2025, 12:24 PM UTC

   [Canada Region] The engineers are working on a hot fix for the web SSO login issue, as well as user profile viewing and searching.

3. identified Oct 28, 2025, 01:20 PM UTC

   [Canada Region] User profile viewing and searching is fixed. Engineers are working on fixing the SSO login issue.

4. monitoring Oct 28, 2025, 02:39 PM UTC

   [Canada region] The fix for SSO login on web browser has been deployed. Users are now able to login to their Hypercare accounts.

5. resolved Oct 28, 2025, 03:50 PM UTC

   The incident is resolved and closing this incident.

Read the full incident report →