Singlewire Software had 29 outages in the last 2 years totaling 19h 23m of downtime — averaging 1.2 incidents per month.
There were 29 Singlewire Software outages since November 27, 2024 totaling 19h 23m of downtime. Each is summarised below — incident details, duration, and resolution information.
We have identified an issue where creating a conference call PIN fails. We have identified a fix and should be deploying that fix shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Tasks set up to run on a schedule, namely Scheduled Notifications and Phone Gathering jobs, are non-functional at this time. We've identified the issue and are preparing a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Singlewire has noticed a temporary downgraded performance due to backend query processes with law enforcement databases. This has since been resolved.
We have received reports of Fusion servers reporting as disconnected. We are investigating to determine the cause of the disconnected status and what, if any, impact this might have on notification reliability.
We have identified an issue with the service that reports on server state and are working to resolve it. Notification deliverability is not affected.
After restarting the affected service, we observe that affected servers are now reporting as being connected. We will continue to monitor for any abnormalities.
After restarting the affected service, we observe that affected servers are now reporting as being connected. We will continue to work to identify the root cause and prevent similar future issues.
The service that determines Fusion server health uses a distributed data store to track the state of each Fusion server in a way that is meant to be resilient to loss of a single node in our system. This service was configured differently from other services in which we use the same technology, and in such a way that the system required data to match on every node in order to process it. As part of a normal system maintenance operation, we replaced several nodes in our system on November 27, including the ones holding this Fusion server health data. Because of this misconfiguration, the system temporarily stopped processing Fusion server health data. To solve the immediate problem, we restarted the service, which put it back into a good working state. We are also planning a longer-term fix correcting the configuration such that future losses of one node cannot result in the same problem occurring, either in this service or other future services using the same technology.