Convex Outage History

Started Mar 04, 2026, 05:28 PM UTC· resolved Mar 04, 2026, 06:28 PM UTC

1. investigating Mar 04, 2026, 05:28 PM UTC

   We are currently investigating an issue leading to elevated error rates on customer backends

2. monitoring Mar 04, 2026, 06:14 PM UTC

   We've identified the issue and remediated the problem. We're monitoring before we declare the all clear.

3. resolved Mar 04, 2026, 06:28 PM UTC

   Incident resolved. We're very sorry for the impact on your projects. Our team will be publishing a detailed postmortem soon.

4. postmortem Mar 05, 2026, 06:14 PM UTC

   Convex had downtime for some customers caused by an anomalous load spike that pushed an internal service into a load-shedding mode, which then triggered an unexpected panic in a caching library we were using. Specifically the load triggered two queue management algorithms: CoDel which proactively drops requests to keep queues small, and adaptive-LIFO which dequeues in reverse order to avoid wasting time on old requests. These are both rather subtle algorithms that large services use to avoid congestion collapse under high load or attack. The panic in the caching library was just a bug that depended on both these algorithms simultaneously. We've made some changes as a result of this incident but the key lesson is that services should try to avoid switching logical behavior during high load. When systems are stressed switching to infrequently-used codepaths can often make matters worse. We're now going to be proactively triggering CoDel and adaptive-LIFO at steady state to ensure that we're exercising this worst-case flow at all times. We apologize to our customers for impacting your products and services. We’re focusing intensely over the next few weeks on hardening our systems to prevent issues like this from happening again.

Read the full incident report →

Started Feb 25, 2026, 01:43 AM UTC· resolved Feb 25, 2026, 01:47 AM UTC

1. identified Feb 25, 2026, 01:43 AM UTC

   A small subset of customers are currently experiencing downtime while we reprovision some hosts. We are working on restoring service.

2. resolved Feb 25, 2026, 01:47 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 10, 2026, 01:26 AM UTC· resolved Feb 10, 2026, 02:01 AM UTC

1. identified Feb 10, 2026, 01:26 AM UTC

   A misconfiguration caused many custom domains to stop working with a TLS error. We are rolling out a fix.

2. monitoring Feb 10, 2026, 01:31 AM UTC

   All hostnames have been reconfigured and should be available again.

3. resolved Feb 10, 2026, 02:01 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 07, 2026, 02:50 AM UTC· resolved Feb 07, 2026, 03:06 AM UTC

1. identified Feb 07, 2026, 02:50 AM UTC

   We’ve identified an overloaded database that’s causing intermittent errors in parts of the Convex Dashboard and CLI. Live traffic to Convex deployments is unaffected. We’re scaling the affected database now and will share another update shortly.

2. monitoring Feb 07, 2026, 03:03 AM UTC

   Error rates are back to baseline. We’re monitoring for stability, but the incident appears resolved.

3. resolved Feb 07, 2026, 03:06 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 05, 2026, 12:28 AM UTC· resolved Feb 05, 2026, 12:37 AM UTC

1. identified Feb 05, 2026, 12:28 AM UTC

   We've identified an issue temporarily blocking redirects for authentication on some non-pro instances. We're actively working on a fix and expect to have this resolved soon.

2. resolved Feb 05, 2026, 12:37 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 28, 2026, 10:40 PM UTC· resolved Jan 29, 2026, 01:09 AM UTC

1. investigating Jan 28, 2026, 10:40 PM UTC

   Some customers are currently unable to push new code without hitting an OptimisticConcurrencyControlFailure error. We're investigating and working on resolving.

2. identified Jan 28, 2026, 10:51 PM UTC

   We've identified a fix and are rolling it out to a subset of customers before deploying more widely

3. resolved Jan 29, 2026, 01:09 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 27, 2026, 04:30 AM UTC· resolved Jan 27, 2026, 04:30 AM UTC

1. resolved Jan 27, 2026, 04:43 AM UTC

   A configuration change in our function runner service caused error rates to increase across all Convex deployments between 8:20pm PT and 8:35pm PT. This change has been rolled back, and all deployments should be operating normally.

Read the full incident report →

Started Jan 17, 2026, 12:09 AM UTC· resolved Jan 17, 2026, 02:03 AM UTC

1. investigating Jan 17, 2026, 12:09 AM UTC

   Text and vector search queries are failing for some deployments

2. monitoring Jan 17, 2026, 01:05 AM UTC

   The issue has been mitigated and we are rolling out a permanent fix.

3. resolved Jan 17, 2026, 02:03 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 09, 2026, 12:51 AM UTC· resolved Jan 09, 2026, 01:40 AM UTC

1. investigating Jan 09, 2026, 12:51 AM UTC

   We're currently investigating and monitoring increased error rates to/from database.

2. resolved Jan 09, 2026, 01:40 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 21, 2025, 10:15 PM UTC· resolved Nov 21, 2025, 10:22 PM UTC

1. investigating Nov 21, 2025, 10:15 PM UTC

   A small subset Convex instances are experiencing elevated errors rates. The Convex team is investigating.

2. identified Nov 21, 2025, 10:19 PM UTC

   The issue has been identified and the team is implementing a fix. Some affected deployments are back online

3. resolved Nov 21, 2025, 10:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 14, 2025, 09:29 PM UTC· resolved Nov 15, 2025, 12:42 AM UTC

1. investigating Nov 14, 2025, 09:29 PM UTC

   Some calls to AWS to deploy new versions of customer code are failing. Provisioning new preview deployments is also impacted. Live production traffic is unaffected.

2. investigating Nov 14, 2025, 09:29 PM UTC

   We are continuing to investigate this issue.

3. monitoring Nov 14, 2025, 11:05 PM UTC

   We have deployed a fix and error rates are falling.

4. resolved Nov 15, 2025, 12:42 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 10, 2025, 06:07 PM UTC· resolved Nov 10, 2025, 06:07 PM UTC

1. resolved Nov 10, 2025, 06:52 PM UTC

   Some customers were unable to create Convex projects between 10:07am and 10:47am pacific, this has since been resolved.

Read the full incident report →

Started Nov 05, 2025, 10:37 PM UTC· resolved Nov 05, 2025, 10:53 PM UTC

1. investigating Nov 05, 2025, 10:37 PM UTC

   Customers on one of our database clusters are experiencing elevated latency and error rates. We're investigating the cause.

2. monitoring Nov 05, 2025, 10:47 PM UTC

   Service was restored within 10 minutes to the affected deployments. The Convex team is monitoring the situation

3. resolved Nov 05, 2025, 10:53 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 25, 2025, 01:15 PM UTC· resolved Oct 25, 2025, 05:34 PM UTC

1. investigating Oct 25, 2025, 01:52 PM UTC

   We are currently investigating this issue.

2. investigating Oct 25, 2025, 02:10 PM UTC

   Identified an issue. Remediating. Pro customers have been remediated fully. Continuing to monitor

3. resolved Oct 25, 2025, 05:34 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 07:54 PM UTC· resolved Oct 21, 2025, 12:58 AM UTC

1. investigating Oct 20, 2025, 07:54 PM UTC

   Our upstream authentication provider is affected by the AWS us-east-1 outage. This is currently impacting the ability for Convex users to login to the Convex dashboard and command line to manage projects. This does not affect your backend traffic.

2. investigating Oct 20, 2025, 10:05 PM UTC

   We are continuing to investigate this issue.

3. resolved Oct 21, 2025, 12:58 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 07:14 PM UTC· resolved Oct 21, 2025, 03:39 AM UTC

1. identified Oct 20, 2025, 07:14 PM UTC

   We're seeing note action availability problems due to AWS Lambda issues.

2. monitoring Oct 21, 2025, 03:23 AM UTC

   Issue should be resolved. Monitoring to ensure things are working.

3. resolved Oct 21, 2025, 03:39 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 02:48 PM UTC· resolved Oct 20, 2025, 10:00 PM UTC

1. investigating Oct 20, 2025, 02:48 PM UTC

   Usage information can't be queried at the moment; we believe this is caused by a new round of AWS issues.

2. identified Oct 20, 2025, 07:20 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Oct 20, 2025, 10:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 07:22 AM UTC· resolved Oct 20, 2025, 09:38 AM UTC

1. investigating Oct 20, 2025, 07:22 AM UTC

   Many databases are currently broken due to an AWS outage in us-east-1

2. monitoring Oct 20, 2025, 09:29 AM UTC

   AWS has mitigated the issue and services are recovering.

3. resolved Oct 20, 2025, 09:38 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 03, 2025, 09:36 AM UTC· resolved Oct 03, 2025, 09:51 AM UTC

1. investigating Oct 03, 2025, 09:36 AM UTC

   Project creation and the dashboard are currently down

2. resolved Oct 03, 2025, 09:51 AM UTC

   Project management was unavailable for approximately 15 mins after a database cluster was removed from circulation following a series of upgrades we're performing. Projects were otherwise running and serving traffic successfully during this period.

Read the full incident report →

Started Oct 02, 2025, 05:23 PM UTC· resolved Oct 02, 2025, 05:23 PM UTC

1. resolved Oct 02, 2025, 09:15 PM UTC

   A bug in our downstream infrastructure provider caused a database cluster to deadlock when it was being resized.

Read the full incident report →

Started Sep 20, 2025, 12:18 PM UTC· resolved Sep 20, 2025, 01:15 PM UTC

1. investigating Sep 20, 2025, 12:37 PM UTC

   We are currently investigating this issue.

2. monitoring Sep 20, 2025, 12:57 PM UTC

   An unexpected traffic pattern overloaded some of our services, causing intermittent unavailability across Convex instances. We've added extra capacity and are monitoring to ensure that the system is stable.

3. resolved Sep 20, 2025, 01:15 PM UTC

   This incident has been resolved.

4. postmortem Sep 20, 2025, 08:42 PM UTC

   From around 5:18am to 5:54am Pacific \(12:18pm to 12:54pm UTC\), Convex had a 36 min period of intermittent downtime that affected all Convex services. The specific issue was a cascading failure in our traffic layer. We had a traffic node \(Caddy\) run out of memory due to an unforeseen load spike and instead of just being restarted/replaced this node was marked as permanently down by our container management layer \(Nomad\) which led to the issue propagating to all traffic servers. Since the incident we've more than doubled the size of our traffic layer, fixed the failover behavior which led to nodes staying failed after OOMing, and will be investigating alternative traffic services. As always data was safe during this incident but we really apologize for the availability impact during that time period.

Read the full incident report →

Started Sep 08, 2025, 11:44 PM UTC· resolved Sep 09, 2025, 12:32 AM UTC

1. investigating Sep 08, 2025, 11:44 PM UTC

   We're experiencing sporadic connection drops in Convex's project management service that may require retries to deploy projects, load the dashboard, etc.

2. resolved Sep 09, 2025, 12:32 AM UTC

   This incident has been resolved.

Read the full incident report →