Singlewire Software incident

Notification Events on On-Premises Servers Unable to Reach Cloud

Major Resolved View vendor source →

Singlewire Software experienced a major incident on April 4, 2024 affecting Android Push Notifications and iOS Push Notifications and 1 more component, lasting 1d 1h. The incident has been resolved; the full update timeline is below.

Started
Apr 04, 2024, 04:44 PM UTC
Resolved
Apr 05, 2024, 06:19 PM UTC
Duration
1d 1h
Detected by Pingoru
Apr 04, 2024, 04:44 PM UTC

Affected components

Android Push NotificationsiOS Push NotificationsEmail NotificationsPhone Call NotificationsSMS NotificationsWebEx Teams NotificationsMicrosoft Teams

Update timeline

  1. investigating Apr 04, 2024, 04:44 PM UTC

    InformaCast Fusion customers may experience issues with certain on-premises notifications which can cause those notifications to not correctly send to cloud recipients. Follow this link for more information: https://support.singlewire.com/s/article/No-notification-history-entry-after-upgrade-to-version-14-22-1 Investigation We’re investigating the part of the on-premises server responsible for handling events related to broadcasts started on those servers. Mitigation We recommend customers who have updated to 14.22.1 to roll back to their previous server version to ensure notifications will reach the cloud correctly. If you have not yet moved to 14.22.1, we recommend that you do not upgrade. Next Steps We will provide an update within the hour with additional details on the impact and any recommended actions to take.

  2. identified Apr 04, 2024, 05:43 PM UTC

    We have identified the issue and are implementing a fix. Mitigation We recommend customers who have updated to 14.22.1 to roll back to their previous server version to ensure notifications will reach the cloud correctly. If you have not yet moved to 14.22.1, we recommend that you do not upgrade. Next Steps We will provide an update within three hours or when we know when an updated server version will be available.

  3. identified Apr 04, 2024, 08:46 PM UTC

    We've implemented a fix and are in the process of validating it before publishing a new version of the Fusion server. Next Steps We'll update again when we have an estimated time for delivery for the fix.

  4. monitoring Apr 04, 2024, 10:13 PM UTC

    We are completing final tests on the fix, and we intend to have a new server version available for download midday CST tomorrow, 4/5.

  5. resolved Apr 05, 2024, 06:19 PM UTC

    New fusion server versions are now available for all customers with the fix included. If you still have servers on version 14.22.1, we recommend that you upgrade to 14.22.2 as soon as you can to ensure reliability of your notifications. This incident is resolved.

  6. postmortem Apr 08, 2024, 05:07 PM UTC

    As part of code changes that went out in our 14.22.1 on-prem server release, we made some performance optimizations to our on-prem to cloud notification workflow by caching some necessary configuration in memory. For some servers, the cached data did not completely match the data that was stored in the database, leading to errors attempting to send on-prem initiated notifications to the cloud. To rectify this, we made changes to the logic we use to check the correctness of the configuration on startup, as well as only caching the configuration in memory once we have read it from the database once. If you are currently running Fusion Server version 14.22.1, please upgrade to Fusion Server version 14.22.2 to receive this fix. Customers running prior versions were not affected by this issue.