Check Outage History

4/28 — Resolution Update and RCA Impact and duration: From approximately 1:05 AM PT to 1:44 AM PT on Tuesday, April 28 (~40 minutes), API endpoints across the platform returned elevated error rates and Console pages were intermittently unavailable. Health checks returned to green at 1:44 AM PT and the system has been stable since. What happened: Two scheduled background jobs ran simultaneously in the early-morning window and pushed our cache layer past its memory limit. Once the cache filled, normal cache writes began failing globally, which surfaced as the broad API degradation. Today's root cause is distinct from the SEV-2 on Friday, 4/24, which was driven by an inbound webhook volume spike from a banking partner. We traced today's failures directly to cache memory pressure, not webhook volume. Going forward: We are taking immediate steps to prevent recurrence, including reducing the load these scheduled jobs place on the cache layer and improving how the system handles cache pressure so it degrades gracefully rather than producing errors. We are also strengthening monitoring on cache health and related signals so this class of issue is caught before it impacts API availability.

This issue has been resolved. This morning Check experienced a database overload from increased webhook traffic that spiked for a a period of time on the morning of 4/24, and surfaced as high latency and error count. The team identified the issue, resolved the immediate database load concerns and restored functionality. The Check engineering team will be following up promptly with a resolution to prevent recurrence.

We identified and resolved the cause of elevated latency and degraded performance. A newly enabled internal process for synchronizing tax filing data generated significantly higher database load than expected, impacting API response times from ~9:15am - 9:38am ET. The feature flag has been reverted and system performance has returned to normal.

All FedACH systems are operating normally again, as reported on the Fed's own status page here: https://www.frbservices.org/app/status/serviceStatus.do. We do not anticipate any further impact and all of our systems are operating normally. Marking this incident resolved.