Voxtelesys had 12 outages in the last 2 years totaling 95h 23m of downtime — averaging 0.5 incidents per month.
There were 12 Voxtelesys outages since August 1, 2024 totaling 95h 23m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are investigating an upstream internet provider who is affecting connectivity for some customers.
The upstream providers issue has been mitigated. We have reported the issue and are awaiting response. We will continue to monitor connectivity in the DFW DC.
This incident has been resolved.
A server node in our SLC Datacenter experienced an unexpected transient hardware fault, which briefly affected service for a subset of customers.
Our Network Operations team immediately intervened, successfully recovering the server and migrating the affected 3CX instances to a healthy node.
We are addressing a localized routing issue affecting internal traffic within the PIT1 data center.
The Network Operations team was able to mitigate the issue by implementing a routing change. We will continue to actively monitor.
Mitigations via routing change have continued to be stable. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
We are currently investigating reports of some users not receiving push notifications on the 3CX Mobile App.
We have identified a connectivity issue affecting some push notification deliveries and have implemented a routing adjustment to bypass the specific external routing irregularity. Immediate testing has shown positive results, and that notifications are delivering successfully. No action is required from your end as we will continue to monitor for stability. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
There have been no further reports of missing push notifications and all tests continue to pass. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
We are investigating monitoring alerts and customer reports of degraded connectivity affecting 3CX instances protected by Cloudflare. Our team is currently coordinating with Cloudflare support to identify the cause and implement a fix.
A workaround is rolling out which should restore connectivity immediately for most users; no manual action is required on your part at this time.
Cloudflare's support has confirmed an issue is occurring with its Magic Transit services. Our engineers have had positive test results for the rerouting of traffic around Cloudflare and are proactively monitoring network traffic to ensure this bypass remains effective while we await a final update from Cloudflare on restoration. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
Cloudflare has reported complete service restoration and officially closed the underlying incident affecting their network. Our network engineering team is maintaining the current network workaround as a proactive measure while we independently verify the stability of their fix. We do not anticipate any further service disruptions related to this event, and we will ensure stability before transitioning traffic back to standard routing.
We are currently investigating an issue with a core networking infrastructure provider that is disrupting access to various services. Our team is actively monitoring the upstream provider's status and their efforts to resolve the connectivity issues. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket via email [email protected]
We have implemented a mitigation strategy by migrating DNS services away from the affected provider and actively clearing the cache on major public DNS resolvers. Service availability is currently being verified with positive test results. Access to the website and portal should now be restored; however, due to the nature of DNS propagation, some users may notice a delay before changes take effect. If you continue to experience issues, please attempt to clear your local browser cache or flush your local DNS. We will continue to monitor the situation closely to ensure stability.
The upstream provider has reported that a fix is being deployed to their infrastructure. To prioritize stability and prevent potential recurrence, we will maintain our DNS services on the current mitigation path until the upstream incident is fully concluded.
The upstream provider has confirmed that their services are operating normally and stability has been restored. We have reverted our DNS services to the primary configuration and verified with positive tests. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
A routine configuration update for a subset of DIDs resulted in those numbers being temporarily unable to accept inbound calls. The vast majority of customers were unaffected. The issue was identified and resolved by reverting the problematic configuration change. The service interruption lasted for 36 minutes for the affected DIDs. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
We are currently investigating reports that our website is not fully loading for some users. Our engineering team is actively working to identify the root cause of this issue.
Our engineering team has implemented a fix for the loading issues, and we are seeing positive test results. We will continue to monitor.
Our implemented fix has remained stable over the past several hours. We have verified that web content and related services are functioning normally.
We have received reports of intermittent dropped connections for customers in/around the West Coast. Voxtelesys network engineering is actively investigating the issue, with all available resources dedicated to identifying the scope and resolving the impairment to the network as quickly as possible. Please note our initial investigation indicates the issue is isolated to a specific network peering point, which is affecting customers whose ISPs utilize that connection. Our leadership team is fully engaged, with all network events automatically brought to their attention. We recognize the critical importance of reliable communication, and restoring full service is our top priority.
Our network engineering team, in collaboration with upstream transit providers has determined the root cause was external to the Voxtelesys network and our direct providers. The issue stemmed from a transit provider utilized by some affected customers' ISPs, which has now implemented the necessary routing change on their end. Our internal tests show positive results, and connectivity has stabilized. Our team will continue to actively monitor the network to ensure full service restoration and stability for all impacted customers. Please do not hesitate to contact support via ticket if you have questions or concerns. You can create a ticket on our portal: https://portal.voxtelesys.net/support-tickets
Following our last update, continued monitoring has confirmed that all services have remained stable. A final root cause analysis confirmed the issue was external to Voxtelesys. The source was a network event within Arelion (AS1299). While our IP transit partner, Lumen, took immediate steps to reroute traffic and mitigate the impact, the full resolution was implemented by Arelion’s engineers who corrected the routing on their network. This final corrective action was facilitated by engagement from their direct customers, such as the local ISPs of those who were impacted.
Our Operations team identified an elevated failure rate affecting messaging MO and DLR webhook deliveries. Upon detection, our Engineering team was promptly engaged and implemented a fix. Webhook deliveries are now completing successfully. We will continue to monitor the situation to ensure continued stability.
Monitoring continues to show stability. This incident has been resolved.
We've identified an issue that may have impacted faxes sent through our eFax service between at 9:38 AM to 12:44 PM CST today. Some faxes may have been returned with the reason “Failed to Deliver - Error: Internal Failure”. A fix has been successfully implemented. If your faxes were impacted, please resend them at your convenience. We will continue to monitor. Thank you for your patience and understanding.
The incident has been resolved, and successful eFax delivery rates returned to normal. Thank you for your patience as we worked to address the issue.
Voxtelesys NOC was alerted of a downed circuit affecting our DFW datacenter. The issue was isolated to a single Tier-1 upstream transit provider, which was automatically withdrawn from routing. Within 30 seconds, alternative IP transit providers successfully rerouted the affected traffic. We have an active trouble ticket with the implicated IP transit provider who is suspecting a potential fiber cut in Mckinney, TX. We are continuing to monitor.
Further investigation showed a fiber cut which has now been spliced.