[Europe] Offline access-points for a subset of customers

Update timeline

1. identified Jul 05, 2023, 02:09 PM UTC

   Dear customers, We have identified that a subset of transmitters are offline. Impacted customers are using the endpoint ap-eu.vusion.io:64109 Our cloud engineers are working to identify possible fix to bring them back online. On behalf of SES-imagotag, the whole team would like to apology for the impact it may have.

2. identified Jul 05, 2023, 03:33 PM UTC

   Dear customers, We are experiencing major timeout and subset of APs are not able to reach our endpoint. It seems to be linked to an on-going incident at Azure: https://azure.status.microsoft/fr-fr/status As stated: "Customers in West Europe may experience packet drops, timeouts and increased latencies in responses. Automated systems are balancing load and rerouting traffic, but customers may still experience degradation. We recommend that customers that are experiencing impact invoke their BCDR plans and fail over to another region if possible." Our Cloud engineers are working to find a fix and to mitigate the issue. Please accept our apologies the whole team would like to apology for the impact it may have.

3. identified Jul 05, 2023, 05:10 PM UTC

   Dear customers, Our engineers are still working to fix this issue and to bring impacted access-points back online. Please accept our apologies for this incident.

4. monitoring Jul 05, 2023, 05:43 PM UTC

   Dear customers, Our engineers have applied a workaround procedure and our metrics are showing that access-points are now back online. The incident at Azure is still opened but our service is stable. We will continue to monitor to ensure that the service has fully recovered. On behalf of SES-imagotag, please accept our apologies for this incident and its impact. Please be assured that we are working continuously to improve our Cloud.

5. resolved Jul 05, 2023, 08:38 PM UTC

   Dear customers, The incident is now resolved. The solution that has been implemented has proven effective, transmitters are back online and stable. All transmissions are going through. Please be assured that no data or update have been lost. The whole team would like to apology for the inconveniences. A root cause analysis will be provided in the coming days.

6. postmortem Jul 06, 2023, 02:51 PM UTC

   Dear customers, On the 6th of July 2023, we conducted an exceptional maintenance on a single endpoint to enhance long-term stability, following the 1-hour disconnection of transmitters for a subset of customers on the 30th of June and 5th of July. However, during the maintenance, a subset of customers experienced offline transmitters, resulting in failed transmissions. Specifically, the transmitters targeting _ap-eu.vusion.io:64109_ were affected. The maintenance was scheduled to last between 30 minutes to 1 hour and was communicated through our support channels. The maintenance was initiated at 12:30 UTC. Unfortunately, we encountered network issues during the operations, leading to timeouts and failed operations. These network issues were caused by an Azure incident \([https://azure.status.microsoft/fr-fr/status/history/,](https://azure.status.microsoft/fr-fr/status/history/,) tracking id: FVHB-188\). The connectivity was unstable, and most transmitters were unable to maintain a stable connection. Our cloud engineers swiftly implemented workarounds and made configuration changes to restore service. At 17:00 UTC, we identified a restriction that caused the system to become unstable when a very high number of transmitters attempted to connect. Our engineers promptly adjusted the configuration and deployed the changes to the endpoint. By 17:40 UTC, all transmitters were back online, and the connections were stable. All operations resumed successfully. No data or transmissions were lost, and no actions were required from our customers. To improve the overall reliability of our platform and mitigate the risk of similar incidents in the future, we have taken the following steps: 1. We have monitored and confirmed that the implemented version is significantly more stable and capable of handling exceptionally high loads. 2. Our cloud engineers will apply this new configuration to all endpoints. 3. We will distribute transmitters to other endpoints to reduce the load on any single endpoint. 4. We have updated our support and deployment procedures to enable faster response times in similar incidents. On behalf of SES-imagotag, the entire Cloud team would like to apologize for any inconvenience caused. Please be assured that we are continually working to enhance the quality of our platform and improve the user experience. Thank you for your understanding and continued support.