Convex Outage History

We had elevated latencies on a single database that required us to temporarily shed load while resizing and reparenting the db. This lead to a period of elevated latencies for all instances on that database followed by downtime for a smaller subset of instances while we performed the maintenance operation. This issue should not occur under normal circumstances and was due an underlying database problem that required manual intervention to resolve. We apologize for the impact to those impacted!

At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.

Convex had downtime for some customers caused by an anomalous load spike that pushed an internal service into a load-shedding mode, which then triggered an unexpected panic in a caching library we were using. Specifically the load triggered two queue management algorithms: CoDel which proactively drops requests to keep queues small, and adaptive-LIFO which dequeues in reverse order to avoid wasting time on old requests. These are both rather subtle algorithms that large services use to avoid congestion collapse under high load or attack. The panic in the caching library was just a bug that depended on both these algorithms simultaneously. We've made some changes as a result of this incident but the key lesson is that services should try to avoid switching logical behavior during high load. When systems are stressed switching to infrequently-used codepaths can often make matters worse. We're now going to be proactively triggering CoDel and adaptive-LIFO at steady state to ensure that we're exercising this worst-case flow at all times. We apologize to our customers for impacting your products and services. We’re focusing intensely over the next few weeks on hardening our systems to prevent issues like this from happening again.