P2 - Since the 7th of August 6pm CET, our presence and occupation reports are not updated

Update timeline

1. identified Aug 08, 2024, 12:41 PM UTC

   Since the 7th of August 6pm CET, our presence report is not updated with any occupation or presence events. Our technical team has identified the root cause and is correcting it. We keep you informed.

2. identified Aug 08, 2024, 01:20 PM UTC

   We are continuing to work on a fix for this issue. Presence and occupation events are still not recorded in the reports.

3. identified Aug 08, 2024, 01:58 PM UTC

   We are still working on a fix. Thanks for your patience.

4. identified Aug 08, 2024, 03:44 PM UTC

   We are continuing to work on resolving this problem. Impact is on statistics only: the agents' presence and occupation events are not recorded since Aug 7th at 6pm CET. Therefore you may see inconsistencies in the presence and occupation reports, as well as on the presence indicator in the Supervision. However there is no impact in handling conversations nor in conversation volume. We are sorry for the inconvenience caused.

5. identified Aug 09, 2024, 08:20 AM UTC

   We have finished an upgrade that ran all night. We are currently validating that the presence data in the reports is calculated again. We are sorry for the inconvenience caused.

6. monitoring Aug 09, 2024, 09:24 AM UTC

   We confirm that the presence and occupation data are being recorded again. We are now working on resynchronising any user account that require it. As no presence or occupation events were recorded during the incident, reports linked to this data may display inconsistencies during this period.

7. resolved Aug 09, 2024, 02:46 PM UTC

   After a monitoring period, we confirm that the incident has been resolved. The presence and occupation data are once again being recorded. However, during the period of the incident (from Wednesday 7 August 6:03pm CET to Friday 9 August 10:57am CET) the presence and occupation events were not recorded. Therefore, you may notice inconsistencies in the reports related to these indicators. On behalf of iAdvize, we apologise for any inconvenience caused.

8. postmortem Aug 14, 2024, 02:48 PM UTC

   ## **Incident** From the 7th of August 8:04 pm CEST to the 9th of August 9:38 am CEST, the presence reports were no longer updated. We hit a size limit issue on a field in one data table that prevented any update. Thus, the state of the agents remained unchanged \(connected or disconnected\) until the incident was resolved. Unfortunately, we can’t recover the state of the data over the period of time of the incident. ## **Resolution** We changed the field data type in the presence table and its format to unlock the size limit and allow for durable storage of presence events. Despite the return to normal on August 9th, we noticed some desynchronization in the agents' states. A manual cleaning of agent presence was required to solve this. Some isolated cases may remain for some agents. If you are facing this situation, please recommend your agents to trigger the disconnection/reconnection action to record again the presence events. ## **Actions for the future** * \(In Progress\) Audit of the tables that could face the same issue and run the fix \(format change\) in anticipation * \(Done\) Strengthen probes to be warned in real time when presence events do not flow as expected