Hypercare incident

[US Region only] Reports of higher latency

Major Resolved View vendor source →

Hypercare experienced a major incident on May 29, 2024 affecting Login and Single Sign On (U.S. Region) and Virtual Pager (U.S. Region) and 1 more component, lasting 2d 2h. The incident has been resolved; the full update timeline is below.

Started
May 29, 2024, 12:28 PM UTC
Resolved
May 31, 2024, 02:45 PM UTC
Duration
2d 2h
Detected by Pingoru
May 29, 2024, 12:28 PM UTC

Affected components

Login and Single Sign On (U.S. Region)Virtual Pager (U.S. Region)Messaging (U.S. Region)Notifications and Real-Time Syncing (U.S. Region)File Attachments (U.S. Region)Viewing Who is On-Call (U.S. Region)Code Teams (U.S. Region)Self-serve Scheduling (U.S. Region)Administration and Scheduling (U.S. Region)API & Integrations (U.S. Region)

Update timeline

  1. investigating May 29, 2024, 12:28 PM UTC

    We have received reports of higher latency to perform actions on the U.S. server. We are currently investigating to see the extent of the problems.

  2. investigating May 29, 2024, 12:36 PM UTC

    We are currently noticing that certain actions can take quite a long time. We are currently investigating the root cause.

  3. identified May 29, 2024, 01:02 PM UTC

    The team is continuing to investigate reports of slowdown in services for US-based users.

  4. monitoring May 29, 2024, 04:01 PM UTC

    Slowness experienced across Hypercare services have stabilized and the root cause has been identified. The team will continue monitoring responsiveness of Hypercare services. Intermittent slowdowns may continue to occur as the team continues to conduct tests and make enhancements as part of the resolution.

  5. resolved May 31, 2024, 02:45 PM UTC

    The team has completed their monitoring of the slowness experienced across Hypercare services on May 29, 2024. No new issues arose during the monitoring period after the incident.