Identified unitQ Monitor data processing issue

Update timeline

1. identified Mar 22, 2023, 10:51 PM UTC

   We have identified a data processing issue with unitQ Monitor where some sources such as Zendesk are experiencing a delay of several hours. We are working on a fix, please be patient as we continue work towards a resolution. We will let you know this issue has resolved.

2. identified Mar 22, 2023, 11:00 PM UTC

   We are continuing to work on a fix for this issue.

3. identified Mar 22, 2023, 11:06 PM UTC

   Alerts on Monitors and Saved Searches have been suspended while we investigate this issue.

4. identified Mar 23, 2023, 12:23 AM UTC

   unitQ Monitor is experiencing delays in processing data for some sources, especially Zendesk, and we have suspended Alerts until this issue is resolved to prevent issuing false positive Alerts. We have now identified the root cause of the problem. This issue was accidentally introduced from a system upgrade which resulted in sending twice as much feedback than usual through our pipeline, intermingling brand new feedback with reprocessed feedback, and leading to some delays in processing the new feedback. We have increased capacity in our pipeline to more quickly process this backlog of data. We estimate that it will take at least several hours to process all of this feedback, so we anticipate that this issue should be resolved sometime tomorrow. We will provide regular updates as the situation progresses. Thank you for your patience and feel free to reach out to us with any questions.

5. monitoring Mar 23, 2023, 03:30 PM UTC

   All services are now back to normal, and able to receive, search, and evaluate monitors as normal. Some customers may still observe gaps in historical data for parts of the last 24 hours, and we are continuing with the backfilling operations.

6. resolved Mar 23, 2023, 10:30 PM UTC

   The backfill of data is complete and fully accessible on the platform. Thank you for your patience and understanding.