Hypercare incident

[CA Region only] Unable to Initiate Hypercare Login

Notice Resolved View vendor source →

Hypercare experienced a notice incident on December 20, 2024 affecting Login and Single Sign On (Canadian Region) and Virtual Pager (Canadian Region), lasting 24m. The incident has been resolved; the full update timeline is below.

Started
Dec 20, 2024, 04:49 AM UTC
Resolved
Dec 20, 2024, 05:13 AM UTC
Duration
24m
Detected by Pingoru
Dec 20, 2024, 04:49 AM UTC

Affected components

Login and Single Sign On (Canadian Region)Virtual Pager (Canadian Region)

Update timeline

  1. investigating Dec 20, 2024, 04:49 AM UTC

    There is interruptions on Hypercare new login sessions in CA Region. Users who are logged in already will stay logged in. We're currently investigating the issue.

  2. identified Dec 20, 2024, 05:07 AM UTC

    Our team currently has identified the issue and is currently identifying the best path for resolution. Our team has also identified features that depend on authentication could be affected including virtual pagers and integrations that depend on such.

  3. monitoring Dec 20, 2024, 05:10 AM UTC

    The issue has now been resolved. Authentication, integrations, and our virtual pager is behaving as normal. Our team is currently monitoring.

  4. resolved Dec 20, 2024, 05:13 AM UTC

    This incident is now resolved.

  5. postmortem Dec 20, 2024, 05:32 AM UTC

    ### **What Happened?** An emergency system update was carried out in the Canadian region starting at 22:15 \(EDT\) on December 19, 2024. An issue was detected at 23:22, where the data migration part of the system update resulted in authentication services to become temporarily unavailable. ### **Impact** Any services requiring authentication were temporarily unavailable. This impacted the following Hypercare services: new logins, virtual paging and integrations. For users who were already logged into Hypercare, there were no interruptions to services. ### **Resolution and Next Steps** We have identified key steps to prevent a recurrence: 1. **Additional testing workflows and simulation:** Numerous checkpoints, test cases, and forced stop points were added for both simulation testing and deployment procedures to identify and mitigate any potential for service interruptions. 2. **Enhanced Monitoring:** new automated monitoring deployed to simulation environments to provide more granular information on migration performance . We sincerely apologize for the disruption and remain dedicated to ensuring a reliable, high-performing platform for all users.