elevated error rates on convex for pro instances

Update timeline

1. investigating Mar 12, 2026, 01:17 PM UTC

   We are currently investigating an issue leading to higher error rates

2. monitoring Mar 12, 2026, 01:40 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 12, 2026, 01:55 PM UTC

   This incident has been resolved.

4. postmortem Mar 13, 2026, 03:36 AM UTC

   At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.