Singlewire Software had 26 outages in the last 2 years totaling 662h 4m of downtime — averaging 1.1 incidents per month.
There were 26 Singlewire Software outages since August 1, 2024 totaling 662h 4m of downtime. Each is summarised below — incident details, duration, and resolution information.
We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.
This incident has been resolved.
We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.
This incident has been resolved.
We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.
We are continuing to investigate this issue.
This incident has been resolved.
We're investigating an intermittent issue with SMS message delivery
This incident has been resolved.
Our monitoring system triggered an alert that indicated a potential service disruption to SMS delivery. After investigation, we confirmed that all systems were operating normally throughout this period. The alert was caused by overly aggressive monitor thresholds, not an actual service issue. We are tuning our alerting to reduce false positives.
We have received reports from RapidSOS of elevated errors that may prevent Emergency Calling from working with InformaCast. For additional information, please visit: https://status.rapidsos.com/ We will update this incident when more information is made available.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Visitor Aware is currently down due to issues with our third party provider Cloudflare. They are working to get the issued resolved and we will keep you updated on their progress.
The issue has been identified and a fix is being implemented.
Cloudflare has implemented a fix. We will continue to post updates as we monitor their progress.
Cloudflare appears to resolved their issue but not yet given the all clear. We will continue to monitor their progress.
We are continuing to monitor Cloudflare as they make progress on their issue.
Cloudflare is continuing to make progress on their recovery.
We are still waiting to hear back from Cloudflare that they have resolved all issues. All indications are that everything is back to normal.
The Cloudflare issue has been resolved.
An increase of errors was detected starting shortly after 11:30am CDT. These failures were linked to the ongoing cloud provider outage.
Visitor Aware is currently experiencing extended processing times and inconsistent facial recognition matching due to the ongoing cloud provider outage. We are monitoring the situation and will update when we learn more.
We are monitoring the situation and will update when we learn more.
Our cloud provider is reporting that their service has recovered. While there may be some increased latency, we believe service should be back to normal soon. We'll continue to monitor for the next hour.
This incident has been resolved.
InformaCast Fusion is currently experiencing a service interruption due to a widespread incident reported by our cloud provider. At this time Fusion cannot send SMS messages, place phone calls, or send email. Our engineers are aware of the issue and are continuing to monitor the situation closely. We will update this incident when we learn more.
InformaCast Fusion is now able to send SMS messages and place calls reliably. Email send are succeeding, but inconsistent delivery times are noted. We will update this incident when we learn more.
All Fusion notification channels are responding normally. We will continue to monitor the situation for next hour to ensure there are no lingering issues.
All InformaCast Fusion functionality has been restored and returned to normal.
Visitor Aware is currently experiencing an issue with badges not being automatically printed through the Desktop Print Helper app. This impacts version 2.1.3 of Print Helper and 12.2.6 of the web portal. The team is investigating the issue.
With the release of version 12.2.6 last night, an update to the format of our visitor and student badges caused an error with current and previous Desktop Print Helper Versions (2.1.3 and previous). This resulted in certain dashboard updates to fail, and caused badges to not print automatically when using Desktop Print Helper to process print jobs. This change has been reverted so that Desktop Print Helper continues to work with version 12.2.6. To address this, we will update the badge formatting fix so that it is compatible with previous Desktop Print Helper versions.
This incident has been resolved.
We have identified an issue where some android devices are not receiving push notifications as a recipient of notifications in InformaCast Fusion. The issue is currently still under investigation.
We have swapped out a copy of one of our services, as we believe the issue is transient at this time. Both monitoring and root cause analysis are ongoing. We will resolve the incident when we are confident that Android pushes are no longer failing.
This incident has been resolved.
During our investigation of this issue, we discovered a programming error with our message delivery code, that, in certain instances, caused it to stall indefinitely. We have isolated the issue and are deploying a fix to prevent it from happening again. We apologize for service disruptions caused by this incident.
Users attempting to use the Microsoft Teams integration with InformaCast to send scenarios may experience a problem where dropdowns display options from previous questions in the sending workflow rather than the current question. This prevents the user from selecting a valid response and sending the scenario. We've investigated and currently believe this to be an issue within recent versions of Microsoft Teams clients, not an error within our integration. We'll continue to monitor and investigate, and we'll update here when we have more information.
Our engineers are in correspondence with Microsoft support about this issue. At this time, scenario sending from Microsoft Teams clients is still impacted. We'll update when we know more.
Microsoft has confirmed the unexpected behavior while sending scenarios from their Teams clients is a defect within their system. Because the issue is external to us, we don't have a timeline for this issue to be resolved. Please refer to this Singlewire Knowledge Base article for the most up-to-date status on this issue: https://support.singlewire.com/s/article/Unable-to-send-scenario-from-Microsoft-Teams-because-scenario-question-displays-incorrect-options
We're experiencing an issue where users may be updated by the local user loader unintentionally, which may cause them to be removed from security groups and distribution lists. Our development team is working to identify the root cause and implement a solution. Any customer using the local user loader may be affected by this issue, so we recommend pausing any local user loader automation and holding off on manual updates to load requests. Cloud user loaders (Entra, School Information Systems) are not impacted by this issue. We'll update this incident when we have identified a root cause or within three hours.
We have identified the problem and are implementing a fix. To minimize additional impact until the fix is complete, we are temporarily disabling all local user loaders.
We have identified the root cause of the issue and are testing a fix for it. All local user loaders remain non-functional as a precaution until we have finished the fix. We are working to identify and restore user data that was erroneously changed or deleted.
We have deployed a change that fixes the issue and re-enabled local user loaders. We continue to work towards restoring user data that was erroneously changed or deleted.
Although the issue with the Local User Loader has been corrected, our Support team will be reaching out to affected customers to help restore their data in the next few days.
We are investigating reports of an issue involving WebEx Calling that is preventing new devices from registering. Modification of existing devices will prevent them from initiating scenarios.
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Visitor Aware is currently experiencing an issue with emails not being sent. This includes automatic emails sent from the system. The team is investigating the issue.
We are continuing to investigate issues with Visitor Aware email sends. Our next update to this incident will happen at 4pm central time or when we have identified a solution.
We are continuing to investigate issues with Visitor Aware email sends. Our next update to this incident will happen at 6pm central time or when we have identified a solution.
This incident has been resolved.
We are currently experiencing degraded performance on VisitorAware.com. We are investigating root cause and will update this incident as we learn more.
This incident has been resolved.
We disabled all SFTP file processing to address performance issues, which caused a backlog of files that attempted to process simultaneously during peak usage. This overloaded the database and resulted in timeout errors. To address this, we gradually re-enabled SFTP processing to run during off hours and increased the database size for higher memory and performance.
Notifications sent using a message template or scenario in a non-global site context and initiated via an on-premises trigger such as DialCast, CallAware, Night Bell, or M2M are currently not delivering to cloud or mobile recipients, nor are they delivering to any on-premises recipients that would be activated by a different Fusion server than the initiating server. On-premises recipients will still receive notifications if the activating Fusion server is the same as the initiating server or the notification is using a message template or scenario from a global site context.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
When testing some permissions changes to our site-based permissions model, we failed to account for situations where initiation methods that come from sources other than users, such as an on-premise device, interact with message templates or scenarios assigned to a particular site. In these cases, we should take a permissive approach to initiating the notification rather than disallowing the interaction. We rolled this change back in the short-term to ensure that this workflow would continue to work. We are taking steps to ensure that this use case is covered in our automation as we attempt to reapply this change.
Some bell schedules fired on 2/12/2025 between 10:17 AM CST and 10:24 AM CST may not be present for review in notification history in the InformaCast web console. This did not affect bell schedule activation for any on-premise devices.
Notifications initiated from the web, mobile apps, or any cloud-side inbound integration may have failed to deliver to on-prem recipients between approximately 10:21 AM and 10:33AM on 1/15/2025.
During this window, one of our engineers was executing a scheduled InformaCast deployment. During this deployment, we encountered an unexpected error. The established procedure during such an event is to immediately escalate to a senior engineer to assess the error and its potential impact. This process was not followed and remediation steps were taken instead, which temporarily left InformaCast with too few copies of a critical service for cloud to on-premises notification to function. Steps are being taken to reinforce training around deployment communications, escalating to senior engineers or management when the unexpected occurs, and the importance of following established procedures.
We have identified an issue where creating a conference call PIN fails. We have identified a fix and should be deploying that fix shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Tasks set up to run on a schedule, namely Scheduled Notifications and Phone Gathering jobs, are non-functional at this time. We've identified the issue and are preparing a fix.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Singlewire has noticed a temporary downgraded performance due to backend query processes with law enforcement databases. This has since been resolved.
We have received reports of Fusion servers reporting as disconnected. We are investigating to determine the cause of the disconnected status and what, if any, impact this might have on notification reliability.
We have identified an issue with the service that reports on server state and are working to resolve it. Notification deliverability is not affected.
After restarting the affected service, we observe that affected servers are now reporting as being connected. We will continue to monitor for any abnormalities.
After restarting the affected service, we observe that affected servers are now reporting as being connected. We will continue to work to identify the root cause and prevent similar future issues.
The service that determines Fusion server health uses a distributed data store to track the state of each Fusion server in a way that is meant to be resilient to loss of a single node in our system. This service was configured differently from other services in which we use the same technology, and in such a way that the system required data to match on every node in order to process it. As part of a normal system maintenance operation, we replaced several nodes in our system on November 27, including the ones holding this Fusion server health data. Because of this misconfiguration, the system temporarily stopped processing Fusion server health data. To solve the immediate problem, we restarted the service, which put it back into a good working state. We are also planning a longer-term fix correcting the configuration such that future losses of one node cannot result in the same problem occurring, either in this service or other future services using the same technology.