Vusion experienced a minor incident on April 12, 2024 affecting VUSION Manager - Europe, lasting 1h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 12, 2024, 08:44 AM UTC
Dear customers, Our metrics are showing that a large subset of customers is impacted by unstable connectivity of transmitters. Transmitters are disconnecting and reconnecting regularly. Because of this, transmissions may be failing or delayed. Our engineers are working on the incident. This incident will be updated as events progress. We would like to thank you for your understanding.
- monitoring Apr 12, 2024, 09:27 AM UTC
Dear customers, Our engineers have identified the issue and performed a corrective action at 8:50 UTC. Our metrics are showing that access-points from impacted customers are now coming back online. We will keep monitoring until the full resolution, and would like to thank you for your understanding. On behalf of VusionCloud operational teams, we would like to apologize for the inconvenience.
- resolved Apr 12, 2024, 10:00 AM UTC
Dear customers, The incident is now closed. Our metrics are showing stable connectivity since our engineers' actions, with numbers of online access-points back to the initial values. On behalf of Vusion Group, the whole Cloud team would like to apologize for the incident and to thank you for your understanding.
- postmortem May 01, 2024, 06:11 PM UTC
# **Vusion Cloud Access-Points Instability** **What happened?** On Tuesday, April 9, 2024, from 9 PM to 12 PM UTC, and on Friday, April 12, 2024, from 9 PM to 10 PM UTC, Vusion Cloud Services experienced significant disruptions that impacted a large subset of our customers in Europe. The issue was isolated to transmission modules that failed to maintain a proper connection with Access-Points, causing devices to disconnect and reconnect repeatedly. This partial service outage led to transmissions being delayed or failing at an abnormal rate. **What went wrong, and why? What did we do to resolve the issue?** The incidents were caused by a combination of unusually high traffic, an unforeseen Microsoft event, and a misconfiguration in routing resources. The issue was escalated to Microsoft, and a workaround was found to mitigate the initial incident. In parallel, our engineers identified improvements to enhance resilience against such cases. After a similar situation occurred a few days later, the change was deployed as a hotfix, and operations recovered swiftly. Our continuous monitoring and metrics show that the implemented changes have been effective. **Current Status:** All systems are now back online and normal operations have resumed. Nothing further is required from the customer at this time. **How are we making incidents like this less likely or less impactful?** Our engineers are working to reduce the likelihood of such events and to mitigate their impacts by: * Enhancing configurations and setups to bolster resilience against such disruptions. * Implementing improved monitoring to identify such cases before they reach an impacting state, and fine-tuning alerts for these types of issues. * Continuously improving our processes and playbooks to tackle or mitigate similar issues as soon as possible. We sincerely apologize for any inconvenience caused by this service disruption. We are aware of the impact on your operations, and we are dedicated to continuously improving our services to help prevent similar incidents in the future. On behalf of Vusion Group, we would like to thank you for your support and understanding.