[Europe] Offline access point for a subset of customers

Update timeline

1. monitoring Jun 11, 2025, 02:26 PM UTC

   On June 11th, 2025, at 10:45 AM UTC, a subset of customers experienced a disruption to their Access Points (APs), which intermittently lost connection to VUSION Cloud and were offline for extended periods of time. Impacted customers had their AP targeting apa-eu-2.vusion.io. As soon as the incident was detected, our engineers identified the issue. A routing issue has been identified and a workaround was implemented. The issue was resolved at approximately 12:00 PM UTC. Please note that all operations resumed successfully after the APs went back online and that no data or transmissions were lost, and no actions are required from customers. However, a minor subset of access point are still offline and will need a manual reboot. Customers with offline access points are required to reboot them manually. Be assured that this incident is not related to the Access point update campaign and is closely monitored by our CloudOps team. We appreciate your patience and understanding as we work to resolve this issue. A post-mortem analysis will be published in the coming weeks. On behalf of SES-imagotag, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.

2. resolved Jun 11, 2025, 03:55 PM UTC

   Dear customers, Our metrics are showing that the incident has been resolved. On behalf of VUSION Group, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible. A post-mortem analysis will be published in the coming weeks.

3. postmortem Jul 22, 2025, 10:59 AM UTC

   # What happened? On June 11th around 10:45am utc, a subset of customers began experiencing massive access points becoming offline. The issue was instantly identified by VusionCloud support teams and escalated to our engineering team at 12:00pm utc. Our engineers worked diligently to diagnose and address the issue. Thanks to their efforts, aps started to turn back online by 12:10 PM UTC on the same day. # What went wrong, and why? The incident was caused by an outage in the routing service on the servers targetted by url [apa-eu-2.vusion.io](http://apa-eu-2.vusion.io/). The maximum capacity for parallel connections was reached, and this created instabilities in the access points connectivity. # How did we respond? As soon as the problem was identified our Cloud engineers increased the capacity of the routing servers. Most of the devices could be recovered successfully after restarting the service, while the remainder have been recovered after a manual restart. The Cloud engineers monitored the situation closely and haven't reported further issues with Access Points then. # How are we preventing this in the future? Several actions have been taken to mitigate the impact of similar events in the future: * The size of the servers has been significantly increased. * Additionally, the service scaling capacity has been significantly increased, so the necessary number of servers are added if usage peaks. * The monitoring of Access Points connectivity to V:Cloud has been improved. Thank you for your patience and continued support.