Service Disruption Caused by Excessive Outbound API Requests to Multitracks

Update timeline

1. investigating Nov 09, 2025, 03:04 PM UTC

   Users may experience errors or degraded performance when accessing services reliant on the Multitracks API. We are currently investigating the issue to restore full functionality.

2. identified Nov 09, 2025, 03:25 PM UTC

   The issue has been identified and our team is working on a fix.

3. resolved Nov 09, 2025, 03:42 PM UTC

   We experienced a service disruption caused by an unexpected spike from outbound API requests from our system to Multitracks. This led to instability in both their API and our own services that rely on it. A fix has been implemented on our side, and all services should be back to normal.

4. postmortem Nov 13, 2025, 09:31 PM UTC

   # **Post-Mortem: Service Disruption Caused by Excessive Outbound API Requests to Multitracks** **Date of Incident:** November 9, 2025 **Duration:** ~1 hour **Impacted:** MultiTracks integration; indirect impact to Services and Check-Ins ## **Overview** On November 9, Planning Center experienced a disruption in our MultiTracks integration caused by an internal change to our caching logic. This change unintentionally triggered a large spike in outbound API requests to MultiTracks, which led to their service throttling our traffic. The resulting failures also caused secondary errors in Planning Center services that rely on responses from that API. This issue was rooted in Planning Center’s system behavior, and we take full responsibility for the impact. ## **What Happened** A caching pattern change dramatically increased the number of API requests we sent to MultiTracks. Their service began throttling our traffic, and our retry behavior amplified the load on both systems. This led to elevated errors in the MultiTracks integration and related Planning Center processes. The impact to customers was somewhat minimized because Music Stand apps use local caching, but some experienced errors, slow responses, or missing data. ## **Resolution** We quickly identified the regression and deployed a revert of the caching change. Once reverted, our outbound traffic returned to normal and MultiTracks’ throttling stopped, allowing dependent systems to recover. ## **Root Cause** * A caching change that generated excessive outbound API requests * Retry and timeout behaviors that worsened cascading failures ## **Preventive Actions** 1. Review and tune third-party timeouts to avoid cascading failures 2. Improve graceful degradation strategies for external dependencies 3. Enhance pre-deployment checks for changes affecting outbound request volume ## **Current Status** The MultiTracks integration is stable. We are implementing the above improvements to prevent similar issues and ensure our systems remain resilient even when dependencies become unstable.