Slower parsing and processing of Lists

Update timeline

1. investigating Dec 13, 2023, 07:32 PM UTC

   We are currently investigating issues with list processing on the Orbee Platform. At this time, all requests are queued and stored, but are being processed slower than usual. We will provide another update soon once we have identified details on the problem.

2. identified Dec 13, 2023, 09:32 PM UTC

   We have identified the issue isolated to the database. All data is still being successfully committed, but slow database performance is happening due to some storage cleanup. We will provide another update as the progress of the storage cleanup is completed. We expect it to be completed within the next hour or so. We are also applying database operation optimizations as well to prevent this type of buildup in the future. Another update will be provided soon once these changes are also applied.

3. monitoring Dec 13, 2023, 09:54 PM UTC

   A fix has been applied and now the backlog of tasks is moving fast and should catch up within the next hour or so. We are monitoring the progress to make sure everything gets back on track, and will provide another update soon.

4. monitoring Dec 14, 2023, 01:45 AM UTC

   We are continuing to monitor these fixes in production. We have seen the entire load on the database reduce drastically and are back to performing like normal. The queue continues to process, and some customers will see their lists processed this evening to get everything back to normal and fully completed. We will provide another update as the final lists get processed and verify the state of things continues to look performant.

5. monitoring Dec 14, 2023, 06:30 AM UTC

   List processing of the remaining lists is still occurring. We will continue to monitor and provide another update within the next hour.

6. resolved Dec 14, 2023, 08:39 AM UTC

   All affected lists have been processed and the system is performing normally.