Reportage - Interaction Reports Slow Response Times

Update timeline

1. investigating Nov 08, 2024, 04:20 PM UTC

   We're experiencing performance issues with Reportage that are resulting in slow response times and timeouts and are actively looking into the issue.

2. identified Nov 08, 2024, 04:23 PM UTC

   While getting the Reportage database prepared for an update, an index was removed that causes interactions reports to either take a long time to load or not load altogether. We are attempting to re-add the index but unfortunately, due to the size of the table, this is taking longer than expected.

3. identified Dec 06, 2024, 04:08 AM UTC

   We are continuing to work on a fix to this issue, and greatly appreciate your patience as we do so. After much testing and many failures with other methods, we have decided to segment the data in order to shrink the data set that needs the index re-added, and therefore should speed up the recovery process. When trying to work with the table as a whole, the sheer size of it caused issues with timeouts, out-of-memory errors, and other failures that have prevented us from restoring full functionality. We have also identified ways to add more resiliency to the Reportage application to avoid any future issues like this. We would like to note that this issue appears to only affect some interaction reports. Our testing shows that smaller interaction reports can still be pulled and displayed successfully, but they time out as the size of the report increases. We haven't noticed any similar slowdowns for other types of reports. Again, we thank you for your patience, and we will continue to update this incident as we have more information.

4. monitoring Jan 22, 2025, 03:59 PM UTC

   The database issues affecting interaction reports have been resolved. Reportage is now functioning normally. We are keeping this incident open, as the database must still undergo a major version upgrade. This requires migrating date-time columns to a new format for one more notably large table. We do not expect this process to have a sizable or persistent impact on Reportage performance. However, it is possible there may be intermittent periods during this process where report generation is slower than expected due to database load. We are closely monitoring the upgrade process. Thank you again for your patience.

5. resolved Feb 18, 2025, 08:15 PM UTC

   The database has been fully upgraded and we expect no more issues related to this original incident.