Vusion experienced a minor incident on April 8, 2024 affecting VUSION Manager - Europe, lasting 2h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 08, 2024, 11:36 PM UTC
Dear customers, Our metrics are showing that a large subset of customers are impacting by unstable connectivity of their transmitters. Transmitters are disconnecting and reconnecting regularly. Because of this, transmissions may be failing or delayed. Our engineers are working on the incident. This incident will be updated as events progress. We would like to thank you for your understanding.
- monitoring Apr 09, 2024, 12:47 AM UTC
Dear customers, Our engineers have implementated workarounds and the situation is now stabilized. As of 12AM UTC, transmitters are connected and stable. We will keep monitoring and the incident has been escalated to Microsoft Azure for root cause analysis. Thank you for your understanding.
- resolved Apr 09, 2024, 01:59 AM UTC
Dear customers, The incident is now close. Situation is stable since our engineers' actions. Our metrics are showing stable connectivity. On behalf of Vusion Group, the whole Cloud team would like to apologize for the incident and to thank you for your understanding.
- postmortem May 01, 2024, 06:12 PM UTC
# **Vusion Cloud Access-Points Instability** **What happened?** On Tuesday, April 9, 2024, from 9 PM to 12 PM UTC, and on Friday, April 12, 2024, from 9 PM to 10 PM UTC, Vusion Cloud Services experienced significant disruptions that impacted a large subset of our customers in Europe. The issue was isolated to transmission modules that failed to maintain a proper connection with Access-Points, causing devices to disconnect and reconnect repeatedly. This partial service outage led to transmissions being delayed or failing at an abnormal rate. **What went wrong, and why? What did we do to resolve the issue?** The incidents were caused by a combination of unusually high traffic, an unforeseen Microsoft event, and a misconfiguration in routing resources. The issue was escalated to Microsoft, and a workaround was found to mitigate the initial incident. In parallel, our engineers identified improvements to enhance resilience against such cases. After a similar situation occurred a few days later, the change was deployed as a hotfix, and operations recovered swiftly. Our continuous monitoring and metrics show that the implemented changes have been effective. **Current Status:** All systems are now back online and normal operations have resumed. Nothing further is required from the customer at this time. **How are we making incidents like this less likely or less impactful?** Our engineers are working to reduce the likelihood of such events and to mitigate their impacts by: * Enhancing configurations and setups to bolster resilience against such disruptions. * Implementing improved monitoring to identify such cases before they reach an impacting state, and fine-tuning alerts for these types of issues. * Continuously improving our processes and playbooks to tackle or mitigate similar issues as soon as possible. We sincerely apologize for any inconvenience caused by this service disruption. We are aware of the impact on your operations, and we are dedicated to continuously improving our services to help prevent similar incidents in the future. On behalf of Vusion Group, we would like to thank you for your support and understanding.