Hypercare Outage History

We have received reports of users being routinely logged out of their Hypercare app on iOS devices, impacting their ability to receive app push notifications. Reports have been received from users in both CA and US region. Our Engineering team has identified the potential root cause for this unexpected behaviour and a fix is underway. Users are being encouraged to turn on Safeguard Notifications in Hypercare should they wish to receive a back-up notifications. https://support.hypercare.com/hc/en-us/articles/5376178050445-Safeguard-Notifications-FAQ Users accessing Hypercare on Android devices are not impacted by this issue.

The engineering team is manually updating the secrets to restore the services. Everbridge integration has been restored.

In addition to the previously reported Doppler outage, we have now identified a separate issue affecting virtual paging. This is due to a service disruption with Twilio, a third-party vendor. As a result, virtual paging is currently unavailable and may affect message delivery for some users. You can view the Twilio status here: https://status.twilio.com We continue to monitor both the Doppler and Twilio incidents and will provide updates as services are restored. Thank you for your continued patience.

Automated code escalation has been manually restored. All code team functions are now fully operational.

We have identified the issue. The team is working on creating a fix.

Hypercare services continue to be interrupted in the US region due to ongoing maintenance work. The Engineering Team has identified the issue and a fix is underway.

The issue has been identified and a fix is being implemented.

We have identified issues with user invitation flow on admin portal. Organization administrators may run into issues when inviting new users or re-inviting users to organization. Product team is working on a fix.

The issue has been identified and a fix is under testing.

The Android app version is currently under testing and review. In the meantime, if users are logged out from the account and not logged into any mobile device, they should receive SMS for every message they receive if they have a mobile number on profile. Users can log into web to check messages until the issue is fixed. Users can also add their phone number to their profile following this instruction: https://support.hypercare.com/hc/en-us/articles/24953957201037-How-to-Add-an-Email-Address-and-Mobile-Number-to-Your-Profile

[Web App Only] The on-call schedule for overnight shifts within the client web application may incorrectly display "No one is currently on call," despite the possibility that someone may indeed be on call during this period. Recommendation: In the meantime, we suggest using the Hypercare mobile app to access the on-call schedule. Alternatively, for users opting to use the web app, navigate to the "Schedule" tab located to the left of "Messages," select the appropriate department, switch the month to June 2024, and review the overnight shift schedule for June 30th.

We have identified an issue that have impacted virtual paging services that are tied to on-call schedule roles. Virtual paging services that are tied directly to individual user account(s) are not impacted. External callers attempting to dial the virtual pager number may encounter a timeout error, resulting them in not being able to leave a callback number and/or an associated voicemail. External callers can try to call again until they are able to successfully try again. The Hypercare Engineering team is continuing to investigate the root cause of the issue and will provide further updates as soon as they become available.

The team is continuing to investigate reports of slowdown in services for US-based users.