Vusion experienced a notice incident on October 3, 2024 affecting VUSION Manager - Europe and VUSION Cloud API - Europe, lasting 2h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 03, 2024, 11:49 AM UTC
Dear Valued Customers, We are aware that some customers are experiencing issues with intermittent connectivity due to transmitter instability. This may result in occasional disruptions or delays in transmission. Please rest assured that our engineering team is actively working to resolve the issue as quickly as possible. We will continue to provide updates as progress is made. We appreciate your patience and understanding, and we are committed to restoring stable service as soon as possible. Thank you for your continued support. Sincerely, VusionCloud team
- monitoring Oct 03, 2024, 12:18 PM UTC
Dear Valued Customers, We are pleased to inform you that a fix has been successfully deployed to all affected endpoints. Our teams are actively monitoring the situation to ensure continued stability. We appreciate your ongoing support and patience. Sincerely, The VusionCloud Team
- resolved Oct 03, 2024, 02:39 PM UTC
Dear Valued Customers, We are pleased to inform you that the issue affecting connectivity has been fully resolved. All systems are now stable, and services are operating as expected. Thank you for your patience and continued trust in VusionCloud. Sincerely, The VusionCloud Team
- postmortem Oct 09, 2024, 09:43 AM UTC
## What happened ? On October 3rd, between 8:20 AM UTC and 12:00 PM UTC, a subset of customers experienced multiple access points going offline for up to an hour. This outage caused business disruptions and halted transmissions in stores. Once the issue was identified, the VusionCloud team swiftly took corrective action to restore service and implemented a fix to prevent future disruptions. ## What went wrong, and why? During the execution of a scheduled technical job, some endpoints encountered an issue that led to abnormally high resource consumption. As a result, endpoint performance degraded, eventually causing access points to disconnect due to resource exhaustion. Once an alert was triggered, VusionCloud engineers took immediate corrective action to restore the service. ## How did we respond? Once the incident was detected, our team prioritized restoring system health. The endpoint server was restarted, bringing the access points back online and allowing services to resume normal operation. Afterward, a thorough analysis of logs and metrics was conducted to identify the job responsible for the disruption. The job was disabled to prevent any future occurrences of the issue. ## How are we making incidents like this less likely or less impactful? We are committed to minimizing the likelihood of such events and mitigating their impact. The jobs that caused the disruption have been disabled and will undergo thorough monitoring in their revised versions to prevent future resource exhaustion. We sincerely apologize for any inconvenience this service disruption may have caused. We understand the impact on your operations and remain dedicated to continuously improving our services to prevent similar incidents. On behalf of VusionGroup, we thank you for your continued support and understanding.