Stages not Sending completely from March 27 to April 11th.

Update timeline

1. resolved Apr 11, 2024, 06:00 PM UTC

   Apr 11, 11:00 PDT Resolved - Date Started: March 27th Date Resolved: April 11th Cause of Incident: A Migration/Upgrade to MySql v8 per Amazon AWS requirements occured on March 26th. When the migration happened there was a small bug in the code/query around NULL date values. This caused rows in our AI predictions output to go missing. Any row that had a date or timestamp or a 0 or null or date outside the normal range would cause that row to be removed entirely from the output. This was something the new version of sql was sensitive to that the old version of sql was not. How we will prevent this from happening in future: A Database migration is not a frequent occurrence, something that happens once every 4 years so so. We will do a better job in QAing by involving the Client Success team to QA a few of their largest clients to make sure no anomalies are occurring with the stages and engagement.

2. resolved Apr 17, 2024, 06:04 PM UTC

   Date Started: March 27th Date Resolved: April 11th Cause of Incident: A Migration/Upgrade to MySql v8 per Amazon AWS requirements occured on March 26th. When the migration happened there was a small bug in the code/query around NULL date values. This caused rows in our AI predictions output to go missing. Any row that had a date or timestamp or a 0 or null or date outside the normal range would cause that row to be removed entirely from the output. This was something the new version of sql was sensitive to that the old version of sql was not. How we will prevent this from happening in future: A Database migration is not a frequent occurrence, something that happens once every 4 years so so. We will do a better job in QAing by involving the Client Success team to QA a few of their largest clients to make sure no anomalies are occurring with the stages and engagement.