Singlewire Software incident
On-prem to cloud notification delivery impacted for non-global site contexts
Singlewire Software experienced a major incident on February 12, 2025 affecting On-Premises Notifications, lasting 15h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 12, 2025, 11:36 PM UTC
Notifications sent using a message template or scenario in a non-global site context and initiated via an on-premises trigger such as DialCast, CallAware, Night Bell, or M2M are currently not delivering to cloud or mobile recipients, nor are they delivering to any on-premises recipients that would be activated by a different Fusion server than the initiating server. On-premises recipients will still receive notifications if the activating Fusion server is the same as the initiating server or the notification is using a message template or scenario from a global site context.
- monitoring Feb 13, 2025, 01:08 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 13, 2025, 02:55 PM UTC
This incident has been resolved.
- postmortem Feb 14, 2025, 08:23 PM UTC
When testing some permissions changes to our site-based permissions model, we failed to account for situations where initiation methods that come from sources other than users, such as an on-premise device, interact with message templates or scenarios assigned to a particular site. In these cases, we should take a permissive approach to initiating the notification rather than disallowing the interaction. We rolled this change back in the short-term to ensure that this workflow would continue to work. We are taking steps to ensure that this use case is covered in our automation as we attempt to reapply this change.