Degraded API Availability + Performance

Update timeline

1. investigating Apr 28, 2026, 08:37 AM UTC

   We've identified an event impacting AVI availability and performance inconsistency. Investigation in progress.

2. monitoring Apr 28, 2026, 08:45 AM UTC

   The system is recovering. We are actively monitoring to ensure it remains recovered. Root cause analysis to follow.

3. resolved Apr 28, 2026, 04:01 PM UTC

   4/28 — Resolution Update and RCA Impact and duration: From approximately 1:05 AM PT to 1:44 AM PT on Tuesday, April 28 (~40 minutes), API endpoints across the platform returned elevated error rates and Console pages were intermittently unavailable. Health checks returned to green at 1:44 AM PT and the system has been stable since. What happened: Two scheduled background jobs ran simultaneously in the early-morning window and pushed our cache layer past its memory limit. Once the cache filled, normal cache writes began failing globally, which surfaced as the broad API degradation. Today's root cause is distinct from the SEV-2 on Friday, 4/24, which was driven by an inbound webhook volume spike from a banking partner. We traced today's failures directly to cache memory pressure, not webhook volume. Going forward: We are taking immediate steps to prevent recurrence, including reducing the load these scheduled jobs place on the cache layer and improving how the system handles cache pressure so it degrades gracefully rather than producing errors. We are also strengthening monitoring on cache health and related signals so this class of issue is caught before it impacts API availability.