[RESOLVED] - P1 - No chat notification displayed on some websites

Update timeline

1. resolved Aug 23, 2024, 08:22 AM UTC

   From 11:05am CEST to 2:51pm CEST on the 22nd of August, you may have noticed that no chat notification was displayed on your website. This was caused by a deployement made by our technical team, which introduced a regression. Not all the clients were impacted. The issue was addressed as soon as we noticed it. We are sorry for any incovenience this may have caused.

2. postmortem Sep 06, 2024, 12:32 PM UTC

   **Incident:** On September 2nd, between 11:01 CEST and 13:04 CEST, we experienced an incident impacting the service in charge of the iAdvize engagement \(handling targeting\). ‌ During this timeframe, the display of notifications on our customers' websites and on mobile applications fluctuated between functioning randomly and not being displayed at all. As a result, starting a conversation from Chat / Call / Video / mobile application channels was degraded \(86 min\) or even completely cut off \(37 min\). Social channels were not impacted. ‌ This unavailability of our engagement service occurred because: * After a restart, our mirroring service moved to the same server instance as our engagement service * Due to an unexpected resource usage spike on the mirroring service, the engagement service was left with insufficient resources to scale and run properly ‌ **Resolution** To solve this issue, we manually isolated our mirroring service to different server instance, ensuring the engagement service had enough resources to run properly again. ‌ **Actions for the future** * \(Done\) Isolate our mirroring service away from other critical services * \(Done\) Analyze the causes of the resource increase on our mirroring service, and implement optimizations to reduce its resource usage * \(Done\) Improve alerting alerting in case of network resource issue on server instances