Podio users will face slowness in background job processing like calculations, workflows, exports

Update timeline

1. investigating Feb 26, 2025, 12:38 PM UTC

   We are investigating the issue of Podio users facing slow background job processing.

2. investigating Feb 26, 2025, 01:27 PM UTC

   We are continuing to investigate the issue.

3. investigating Feb 26, 2025, 02:49 PM UTC

   We are continuing to investigate the issue.

4. identified Feb 26, 2025, 03:59 PM UTC

   The issue has been identified and a fix is being implemented.

5. identified Feb 26, 2025, 07:26 PM UTC

   We are still working to restore Podio functionalities. You might continue to see delays in processing of calculations, workflows, large exports and some other long running operations. We will keep providing updates as we proceed towards resolution.

6. identified Feb 26, 2025, 08:06 PM UTC

   We are still working to restore Podio functionalities. You might continue to see delays in processing calculations, workflows, large exports, and some other long-running operations. We will keep providing updates as we proceed toward resolution.

7. identified Feb 26, 2025, 09:24 PM UTC

   We continue to work on fixing this issue. We will keep providing updates as we proceed toward resolution.

8. identified Feb 26, 2025, 10:36 PM UTC

   We are continuing to work on a fix for this issue.

9. monitoring Feb 27, 2025, 04:58 AM UTC

   We have applied a fix to which all operations are back to normal. However, we need to do further updates to achieve a permanent fix for this issue. Till then, we suspect the issues with delayed calculations, workflows, large exports, and heavy API calls might arise again. So we will continue monitoring while we work closely with our database provider to devise and apply a long-term solution.

10. monitoring Feb 27, 2025, 03:26 PM UTC

    We continue actively monitoring the system. Things look positive and we do not see operation delays at the moment.

11. resolved Mar 03, 2025, 04:12 AM UTC

    This incident has been resolved.

12. postmortem Mar 20, 2025, 02:36 PM UTC

    # **Podio search, re-calculations, workflows, real time item updates, chat and notifications event – February 26, 2025** ## **Summary of Impact** On February 26th 2025, Podio customers experienced issues in searching items, calculations on fields, real time item updates and webhook events causing delays and disruptions in workflow execution. ## **Root Cause** On February 26, 2025, at 07:37 EDT, our alert system noticed a sudden increase in activity in one of our queues that handles item searches, updates, and other tasks. The messages in this queue were not being processed, which added to the existing workload. The problem was traced back to a failure in our messaging system, likely caused by a recent software update, which was completed on February 19, 2025. This issue made the queue stop working, leading to delays in processing messages and affecting various services like item searches, calculations, and real-time updates. We worked closely with our third-party database provider to troubleshoot and resolve the issue. ## **Mitigation** After identifying the root cause, the Engineering team took the following steps: * Stopped all incoming traffic to the queue and reverted the AMQ upgrade. All issues were confirmed resolved and fixed as of Feb 26, 2025 23:58 EDT. ## **Next Steps** 1. Conduct load testing of the AMQ upgrade to version 3.13 in our lower environments to identify and fix any bottlenecks before attempting the next upgrade in our production system. 2. Further enhance alerts and monitoring to notify us when our event queue reaches a certain threshold, allowing us to proactively address the issue.