Vusion experienced a major incident on September 19, 2024 affecting VUSION Manager - Europe and VUSION Cloud API - Europe, lasting 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 19, 2024, 09:33 AM UTC
Dear customers, Our metrics are showing that some users may experience issues connecting to the user interface of VUSION Manager and some API calls. Our engineers are investigating. Thank you for your understanding.
- identified Sep 19, 2024, 10:17 AM UTC
We have identified the issue, we are currently working on it.
- monitoring Sep 19, 2024, 10:36 AM UTC
We are currently experiencing an issue affecting user Azure databases (redundant systems). Some databases were either not responding properly or stopped working entirely. Upon investigation, we identified the root cause of the problem and immediately activated our Disaster Recovery (DR) process. The affected workloads have been successfully switched to another region, ensuring minimal disruption to services. Our team is closely monitoring the situation, and we will provide further updates as needed. We apologize for any inconvenience this may have caused and thank you for your understanding
- resolved Sep 19, 2024, 11:34 AM UTC
The issue has been resolved, and all services are now fully operational. Thank you for your patience.
- postmortem Oct 09, 2024, 09:59 AM UTC
## What happened ? On September 19th, starting at 8:51 AM UTC, users began experiencing difficulties accessing Vusion Manager and encountered errors when attempting to call VusionCloud APIs. Although the situation improved after about 15 minutes, the same instability reoccurred several times throughout the hour. To resolve the issue, VusionCloud engineers initiated a failover procedure at 10:15 AM UTC, successfully restoring normal service by 10:20 AM UTC. ## What went wrong, and why? On September 19th, at 8:43 AM UTC, the Users SQL server hosted on Microsoft Azure began experiencing slower-than-usual response times. This slowdown triggered an increase in parallel sessions on the server, which led to a service disruption starting at 8:51 AM UTC. While fluctuations in the number of requests, combined with actions taken by VusionCloud engineers, temporarily improved the situation, the underlying database health issues caused the problem to recur several times over the next hour. ## How did we respond? Automatic alerts were triggered at 9:01 AM, notifying our engineers of the issue. The VusionCloud team immediately prioritized restoring system health. Engineers quickly identified the high database load as the source of the disruption and increased server capacity to reduce the load, allowing requests to be processed again. This action provided a temporary improvement. However, the database slowdown caused the issue to recur shortly after. As the SQL server's automatic failover was not triggered, the VusionCloud team manually initiated a failover to the secondary database at 10:15 AM, which ultimately resolved the problem. ## How are we making incidents like this less likely or less impactful? We are committed to minimizing the likelihood of such events and mitigating their impact. To that end, we have scaled up the backend database, increasing its capacity to better handle unexpected surges in load.