[Europe] Offline transmitters in a subset of customers

Update timeline

1. identified Oct 18, 2022, 05:01 PM UTC

   Dear customers, We have identified that a subset of transmitters are offline since a few hours. Our cloud engineers are working to identify possible fix to bring them back online. On behalf of SES-imagotag, the whole team would like to apology for the impact it may have.

2. monitoring Oct 18, 2022, 06:52 PM UTC

   A fix has been implemented and most transmitters and now back online. Our engineers keep monitoring the situation.

3. resolved Oct 18, 2022, 07:17 PM UTC

   Dear customers, The incident is now resolved. We have implemented a fix and transmitters are back online. All transmissions are going through. Please be assured that no data or update have been lost. On behalf of SES-imagotag, the whole team would like to apology for the inconveniences.

4. postmortem Nov 03, 2022, 02:02 PM UTC

   Dear customers, On the 18th of October 2022, following the release of Europe environment, some transmitters' management services encountered issues getting all transmitters online. As a result, some transmissions may have failed for a subset of customers. Our Cloud engineers identified that transmitters using advanced security were impacted. The transmitters came online very slowly. Some transmitters were not able to get their secure token in a timely fashion and were not able to securely communicate with the transmission service. It was identified that the authentication service was overloaded by transmitters, which got disconnected during the deployment, querying to get authenticated. Around 18:30 UTC, our cloud engineers deployed a fix to allow all transmitters to get their token quickly. At 18:45 UTC, all transmitters were online. All operations resumed. No data or transmission have been lost and no actions were needed from customers. In order to improve the overall reliability and to avoid the occurrence of similar incidents in the future: 1. We monitored and confirmed that the now-implemented version proves to be much more stable and capable of handling even exceptionally high loads, 2. Our Cloud engineers have identified improvements to further increase stability and capacities, 3. Our transmitter development team has identified improvements in the transmitters' behaviour to ensure reliability in case of issues and avoid massive, simultaneous authentication requests, 4. Finally, we have added more extensive load tests during developments to cover such scenarios. On behalf of SES-imagotag, the whole Cloud team would like to apology for the impact it may have had. Please be assured that we are working continuously to improve the quality of the platform and the user experience.