On-Premises Server events may be delayed

Update timeline

1. monitoring Mar 09, 2026, 03:41 PM UTC

   We're experiencing an issue where events from on-premises servers may be delayed from reaching the cloud. This may include events intended to broadcast notifications to the cloud, such as using a phone to activate a DialCast Dialing Configuration. This is also presently impacting notification progress and completion reporting, as well as display of IDNs. For now, to ensure notification delivery to cloud devices, activating notifications from the web application or other cloud devices is encouraged. We've implemented a fix, and we're monitoring the results.

2. monitoring Mar 09, 2026, 06:39 PM UTC

   We are continuing to monitor. At this point, notifications sent from on-premises servers should be received by the cloud and distributed correctly. Events related to on-premises devices such as speakers and IDNs are still being processed; there may be some delay before changes to such devices are visible in the cloud.

3. monitoring Mar 09, 2026, 08:24 PM UTC

   Device updates are still facing delays, but we are starting to see our event processor catch up. We'll continue to monitor this overnight and update this tomorrow, 3/10.

4. resolved Mar 10, 2026, 02:07 PM UTC

   This incident has been resolved.

5. postmortem Mar 10, 2026, 05:52 PM UTC

   On Friday night, we deployed a small infrastructure change that required a reboot of our database. During this reboot, we encountered a bug in our event processor code that caused events not to be processed until service reboot. To compound this, our monitoring platform had invalidated the alarm we had in place to monitor errors with our event processor. This caused us to remain unaware of the issue until Monday morning, where we promptly fixed it by rebooting the service containing the event processors. At that point, the platform needed to catch up by processing all the events that had been building up over the weekend, which completed on Monday night. We will be taking a multi-pronged approach to ensuring this doesn’t happen again: 1. We have fixed our event monitors, and will be testing them every release going forward to ensure they remain operational. We are also following up with our monitoring platform to ensure future monitoring will not be disrupted. 2. We are augmenting our event processing code in the following ways: 1. We have switched to a more robust event processing pattern that is not vulnerable to the bug that caused consumers to stop polling after a database reboot. 2. We have increased the processing throughput such that if this _were_ to happen again, our consumers would catch up much more quickly than they did during this incident. We apologize for service disruptions caused by this incident.