[Europe] ap-eu.vusion.io:64173 - delay in transmissions

Update timeline

1. investigating Jan 09, 2025, 10:01 AM UTC

   Dear customers, A subset of customers using the endpoint ap-eu.vusion.io:64173 are currently experiencing strong delays in image transmissions. Our engineers are actively working on identifying the issue and restore services. Please note that no data has been lost, all transmissions are queued and will be processed eventually. On behalf of VusionGroup, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.

2. identified Jan 09, 2025, 10:21 AM UTC

   Dear customers, The cause for the delay has been identified as a high resource consumption. The server has been restarted and waiting transmissions are being processed. Our teams are closely monitoring the situation.

3. monitoring Jan 09, 2025, 10:41 AM UTC

   Waiting transmissions are still catching up, the majority has been processed already. Our teams are still closely monitoring.

4. resolved Jan 09, 2025, 01:14 PM UTC

   Dear customers, The issue on endpoint ap-eu.vusion.io:64173 has been resolved. All the delays have been resorbed, and all waiting tasks have finished. The endpoint is now healthy again and services resumed to normal. On behalf of VusionGroup, the whole Cloud team would like to apologize for the incident and to thank you for your understanding.

5. postmortem Jan 27, 2025, 05:36 PM UTC

   ## What happened ? Starting on January 9th starting from 4:15 AM UTC, multiple clients experienced access points going offline or delays in transmissions on endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/). This outage caused business disruptions and temporarily halted transmissions in stores. Once the issue was identified, the VusionCloud team swiftly took corrective action to restore service and planned for a long term fix. ## What went wrong, and why? On January 9th, the endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/) started consuming an abnormally high amount of resources. As a result, endpoint performances were degraded, eventually causing access points to disconnect due to resource exhaustion. VusionCloud engineers' analysis showed that the endpoint has been populated with a large number of stores. From 4:15 AM UTC on January 9th, a high activity on many of these stores made the server reach its capacities, which eventually led to the degradation of service. ## How did we respond? As soon as the first alerts were raised, VusionCloud engineers started to investigate the issue and took corrective action to improve the situation. All manual attempts only got limited and temporary results. The server was finally restarted at 8:40 AM UTC, leading the ressources to be cleared and reallocated. The access points came back online and full service was restored from 9:10 am. ## How are we making incidents like this less likely or less impactful? We are committed to minimizing the likelihood of such events and mitigating their impact. As a short term fix, the server was restarted and extra monitoring has been done on this endpoint. As a mid term fix, the server capacity will be increased during January release to make sure a high load does not cause a similar problem in the future. As a long-term fix, the endpoint [ap-eu.vusion.io:64173](http://ap-eu.vusion.io:64203/) has been flagged as prioritary for migrations to our new endpoints, which are more resilients to such loads and that will allow for better load balancing. We sincerely apologize for any inconvenience this service disruption may have caused. We understand the impact on your operations and remain dedicated to continuously improving our services. On behalf of VusionGroup, we thank you for your continued support and understanding.