Podio experienced a notice incident on October 26, 2024 affecting Web and API and 1 more component, lasting 12h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 26, 2024, 09:36 AM UTC
We are currently investigating this issue.
- identified Oct 26, 2024, 12:08 PM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 26, 2024, 01:56 PM UTC
We are still working to optimize Podio's performance.
- identified Oct 26, 2024, 03:49 PM UTC
We are continuing to work on a fix for this issue.
- identified Oct 26, 2024, 03:50 PM UTC
We are continuing to work on a fix for this issue.
- identified Oct 26, 2024, 05:36 PM UTC
We're still troubleshooting the issue to get Podio back as soon as possible.
- identified Oct 26, 2024, 06:08 PM UTC
Following the database maintenance yesterday, we began experiencing spikes in CPU utilisation. Certain complex queries with multiple indexes are causing bottlenecks and negatively affecting performance. We have identified some of these issues and are working on the fixes, but the process is taking longer than anticipated. We apologize for the long wait for the resolution but also thank you for your patience. Be rest assured our team is working on this with highest priority.
- monitoring Oct 26, 2024, 07:36 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 26, 2024, 10:34 PM UTC
Podio is stable and issue marked as resolved.
- postmortem Nov 05, 2024, 07:58 AM UTC
# **Podio Service Event - Oct 26 - 27 2024** ## **Summary of Impact** On October 26th and 27th, 2024, Podio customers encountered performance issues, including login disruptions and general unavailability. This impacted the ability of users to access and utilize Podio, affecting workflows and client engagements. ## **Root Cause** On October 26th, we conducted a scheduled maintenance event from 9:30 AM to 11:30 AM IST to upgrade our database infrastructure. Upon transitioning traffic to the upgraded database post-maintenance, we noticed a substantial increase in CPU utilization alongside an elevated number of active sessions. This surge caused slow performance, preventing many users from successfully logging in. The continued high demand ultimately led to temporary service interruptions. ## **Mitigation** **Immediate Actions:** Our team, in collaboration with third party support, investigated the potential sources of the performance lag and identified that multiple composite indexes in use were contributing to slow query execution times. **Resolution Timeline:** By 1:30 PM IST on October 26th, we optimized several queries to improve response times. We communicated this update on our status page, noting improvements. However, some users experienced issues with category/date filters in Podio apps, which we resolved by 3:00 PM IST on October 27th. **Full Restoration:** By 3:30 PM IST on October 27th, all identified issues were resolved, restoring Podio services to normal operation. ## **Next Steps** * Third Party Collaboration: Continue to work closely with third parties to understand the increased query execution times experienced in the new database environment. * Rollback Strategy: We already have a general strategy in place, but moving forward, we will implement a more robust rollback plan for database upgrades to further reduce risks and ensure seamless experience. * Query Optimisation: We are prioritising ongoing analysis and optimisation of high-demand queries to ensure improved performance and stability. ## **Our Commitment to Service, Reliability and Performance** Thank you for your patience and understanding as we work to enhance the reliability of our services. Please feel free to reach out to our support team via 24/7 active chat support.