TPx Communications incident
Multiple Markets SD-Wan Monitoring Degraded
TPx Communications experienced a minor incident on January 30, 2025 affecting MSx WAN (SD-WAN), lasting 13d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 30, 2025, 04:14 PM UTC
We have identified a degradation of status communications between our SD-Wan devices and one of the Orchestrators that monitors the devices. This only affects customers on Orchestrators VCO-176. Customer traffic is not affected only monitoring and statistics gathering between the edge device and the Orchestrator. Real-Time statistics are not available for customers on this VCO at this time. Our engineering team and vendor are actively troubleshooting this event. If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
- identified Jan 30, 2025, 08:12 PM UTC
Our vendor has escalated this event to their Orchestrator Engineering team for investigation. They are in the process of restarting some services in the Orchestrator that appear to be in a "stuck" condition. They are continuing the debugging of the system to restore it to normal functionality.
- identified Jan 31, 2025, 04:41 AM UTC
Our vendor has engaged additional engineering resources to continue their troubleshooting. The NOC coordinated a meeting with Broadcom Engineering and TPx Engineering to discuss a possible rollback to the previous code. Although still on the table as an option, both Broadcom and TPx have decided the risk of possible loss of services to customers was to high at this time. TPx has asked our vendor to apply as many resources as they have to continue troubleshooting efforts. If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
- monitoring Jan 31, 2025, 01:23 PM UTC
Our Vendor has resolved the issues VCO-176 was experiencing. We are monitoring stability and verifying functions are performing as expected. Customers with access to this VCO should have access and functions as before. If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
- monitoring Feb 03, 2025, 10:10 PM UTC
We continue to work with Broadcom on a few other non-service-impacting issues caused by the Orchestrator upgrade. We are communicating daily with their engineers until all issues have been corrected. If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.
- resolved Feb 13, 2025, 03:34 PM UTC
Our Vendor has completed all non-customer-facing fixes to their software. Our monitoring systems are functioning as expected and have been stable for a week. This event is now being closed. If you feel your service is being impacted, please open a case with our technical support teams at www.tpx.com/support/.