Zomentum Outage History

The Zomentum Payments integration with QuickBooks Online became unavailable after exceeding the monthly API rate limit \(500K calls\) on our Builder App plan. This blocked both QBO data syncing and payment collection for customers with QBO integration enabled. Approximately 350 events were queued during the outage. Timeline * Feb 16, 2026: QuickBooks API rate limit \(500K CorePlus calls\) exceeded. API began returning HTTP 429 with error code 3001 \("ThrottleExceeded"\). * Feb 17, 2026 4:13 AM: Intuit sent automated notification emails to connected QBO customers stating "Zomentum Payments is offline." * Feb 17, 2026: Issue identified as rate limit on Builder App tier. Decision made to upgrade to Intuit App Partner Program for unlimited API access. * Feb 17, 2026: Plan upgrade completed. Throttle lifted. All ~350 stuck events retried and processed successfully. Full functionality restored. Impact * ~350 events stuck in queue during the outage * Payment collection blocked for customers with QBO integration enabled * QBO data syncing paused for all connected customers * Customers received alarming "offline" notifications from Intuit Root Cause Our QuickBooks integration was on the Builder App plan, which has a monthly cap of 500K CorePlus API calls. As usage grew, we hit this limit mid-month, causing Intuit to block all API calls with a "Block All CorePlus for a Builder App" throttle policy. Resolution Upgraded to a higher-tier Intuit plan with increased API limits. All queued events were retried and succeeded once the throttle was lifted. Action Items 1. Implement API usage monitoring and alerting - Set up alerts when QBO API usage approaches the plan threshold \(e.g., at 80% and 90%\) to provide early warning before hitting limits. 2. Add rate limit handling in PaymentsQuickBooksClient - Implement backoff/retry logic and graceful degradation when receiving 429 responses instead of hard-failing. 3. Review API call efficiency - Audit current QBO API usage patterns to identify opportunities to reduce call volume \(batching, caching, eliminating redundant calls\). Lessons Learned * Third-party API rate limits are an operational risk that must be actively monitored. * Critical payment flows should not have hard dependencies on third-party sync operations. * Plan tier limits should be tracked as part of capacity planning.