Hypercare incident

[CA & US Region] Android only - Inconsistent Notifications

Minor Resolved View vendor source →

Hypercare experienced a minor incident on September 1, 2024 affecting Messaging (Canadian Region) and Messaging (U.S. Region) and 1 more component, lasting 52d 22h. The incident has been resolved; the full update timeline is below.

Started
Sep 01, 2024, 10:22 PM UTC
Resolved
Oct 24, 2024, 08:57 PM UTC
Duration
52d 22h
Detected by Pingoru
Sep 01, 2024, 10:22 PM UTC

Affected components

Messaging (Canadian Region)Messaging (U.S. Region)Notifications and Real-Time Syncing (Canadian Region)Notifications and Real-Time Syncing (U.S. Region)

Update timeline

  1. investigating Aug 28, 2024, 05:22 PM UTC

    We have received reports of inconsistent notifications on Android devices. We're currently investigating the issue.

  2. identified Aug 28, 2024, 06:13 PM UTC

    The potential cause for inconsistent push notifications on Android device has been identified, and the fix will need another 2 days to be implemented. In the meantime, Android users are advised to set up the safeguard notifications (FAQ: https://support.hypercare.com/hc/en-us/articles/5376178050445-Safeguard-Notifications-FAQ) or revert to manual escalation if necessary.

  3. identified Sep 01, 2024, 10:22 PM UTC

    Our team has identified the issue and shipped out the fix on the server side. Unfortunately, we will also need to release a new version of the Android app to fully resolve the issue. Our team is aiming to release the new version of Android by September 6th to account for sufficient time for testing and quality assurance purposes.

  4. monitoring Sep 04, 2024, 04:45 AM UTC

    A fix for the root cause of the issue has been implemented. It's backwards compatible and users do not need to update the Android app. We're currently doing more testing and monitoring the performance to ensure notifications are performing as expected.

  5. investigating Sep 05, 2024, 05:08 PM UTC

    We have received reports on Android users not receiving notifications. The engineering team is currently investigating.

  6. identified Sep 05, 2024, 05:09 PM UTC

    The issue has been identified and engineering is deploying a fix.

  7. monitoring Sep 05, 2024, 05:23 PM UTC

    The fix has been deployed and we're currently monitoring the Android notifications behaviour.

  8. monitoring Sep 10, 2024, 06:05 PM UTC

    Android notifications seem stable. We are continuing to monitor the behaviour of Android notifications.

  9. resolved Oct 24, 2024, 08:57 PM UTC

    We've been monitoring the notifications and confirmed that the incident has been resolved.