Upstash Outage History

Started Sep 29, 2026, 03:12 PM UTC· resolved Sep 29, 2026, 03:16 PM UTC

1. resolved Sep 29, 2026, 03:36 PM UTC

   Between 15:12 and 15:16 UTC, we observed a brief increase in load on the connection proxy layer in us-east-1, which may have affected some existing and new connections. The team responded quickly and resolved the issue. All systems are now fully operational.

Read the full incident report →

Started Sep 09, 2026, 12:00 PM UTC· resolved Sep 09, 2026, 12:20 PM UTC

1. resolved Sep 09, 2026, 01:00 PM UTC

   Upstash Vector experienced a brief DNS resolution issue in the US East (us-east-1) region lasting approximately 20 minutes. During this period, new DNS resolution attempts may have failed. Existing connections and clients with cached DNS records were not affected. The issue has been resolved, and DNS resolution is operating normally.

Read the full incident report →

Started Sep 03, 2026, 08:00 AM UTC· resolved Sep 03, 2026, 08:00 AM UTC

1. resolved Sep 03, 2026, 08:29 AM UTC

   We experienced a brief issue affecting access to the Upstash Console. Our team quickly identified the cause and resolved the issue within minutes. The console is now operating normally.

Read the full incident report →

Started Aug 28, 2026, 10:20 PM UTC· resolved Aug 28, 2026, 10:33 PM UTC

1. resolved Aug 28, 2026, 10:56 PM UTC

   Between 22:20 and 22:33 UTC, QStash clients in us-east-1 experienced disruptions related to message persistence. The team applied a fix, and service has been restored.

Read the full incident report →

Started Jul 22, 2026, 08:50 AM UTC· resolved Jul 22, 2026, 10:05 AM UTC

1. identified Jul 22, 2026, 08:50 AM UTC

   We are investigating connectivity issues affecting a subset of Redis databases hosted on Fly.io. The issue is related to an ongoing Fly.io infrastructure disruption in the DFW region (https://status.flyio.net/incidents/n4z2my6qw4sb). We are working to restore availability and will provide updates soon.

2. monitoring Jul 22, 2026, 09:01 AM UTC

   The affected Fly.io-hosted machines recovered at 08:48 UTC, and service availability has been restored. We are monitoring the systems to confirm continued stability.

3. resolved Jul 22, 2026, 10:05 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 16, 2026, 06:30 AM UTC· resolved Jul 16, 2026, 06:30 AM UTC

1. resolved Jul 16, 2026, 08:35 AM UTC

   Partial outage due to http parsing errors. Problem was resolved couple minutes later. If problem continues, refreshing the DNS cache is recommended.

Read the full incident report →

Started Jun 25, 2026, 03:23 PM UTC· resolved Jun 25, 2026, 04:36 PM UTC

1. investigating Jun 25, 2026, 03:23 PM UTC

   We are currently experiencing a partial service disruption affecting Upstash Redis in the following regions: * eu-central-1 * us-east-1 * us-west-1 Our team is investigating the issue and working to restore full service as soon as possible. We will share updates as more information becomes available.

2. monitoring Jun 25, 2026, 03:31 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 25, 2026, 04:36 PM UTC

   This incident has been resolved, we will publish RCA soon.

Read the full incident report →

Started Jun 23, 2026, 04:22 PM UTC· resolved Jun 23, 2026, 05:14 PM UTC

1. investigating Jun 23, 2026, 04:22 PM UTC

   We are currently investigating this issue.

2. monitoring Jun 23, 2026, 04:32 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 23, 2026, 05:14 PM UTC

   This incident has been resolved.

4. postmortem Jun 24, 2026, 12:28 PM UTC

   ### **Summary** On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version. ### **Impact** * Service: QStash \(EU-CENTRAL-1\) * Start: 2026-06-23 16:22 UTC * Resolved: 2026-06-23 17:14 UTC * Duration: ~52 minutes * Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window. ### **Root Cause** We recently deployed an enhancement to our Flow Control feature, designed to improve fairness between independent flow controls when unused global parallelism capacity was available. Under production load, the new behavior introduced a regression that significantly increased resource utilization. The elevated resource consumption caused performance degradation on the affected QStash instamce, impacting all workloads sharing that instance. ### **Resolution** After identifying the regression as the source of the slowdown, we rolled back the deployment to the last known stable version. * Rollback completed: 2026-06-23 19:18:46.84 UTC Following the rollback, system performance returned to normal levels and service stability was restored. ### **Preventive Actions** To reduce the likelihood of similar incidents in the future, we are taking the following actions: * Expand performance and load testing coverage for Flow Control changes. * Add resource utilization regression checks to the deployment pipeline. * Improve monitoring and alerting for abnormal CPU and memory consumption patterns. * Introduce additional canary validation before wider production rollout of scheduler-related changes. We apologize for the disruption and appreciate our customers’ patience while we resolved the issue.

Read the full incident report →

Started Jun 23, 2026, 01:18 AM UTC· resolved Jun 23, 2026, 04:03 AM UTC

1. resolved Jun 23, 2026, 06:05 AM UTC

   Between 01:18 and 04:03 UTC, customers were unable to access running boxes or perform create, read, update, or delete operations. Running containers were not affected and continued operating throughout. The issue has been fully resolved and all box operations are functioning normally. We are taking all necessary measures to prevent this from happening again. We apologize for the disruption.

Read the full incident report →

Started Jun 19, 2026, 09:11 AM UTC· resolved Jun 19, 2026, 11:24 AM UTC

1. resolved Jun 19, 2026, 10:07 AM UTC

   We identified an outbound networking issue affecting a subset of QStash server instances in the EU region between 2026-06-19 09:11 UTC and 2026-06-19 09:45 UTC. During this period, some requests dispatched by QStash may have failed. The issue has been resolved, and QStash is currently operating normally. We will publish a postmortem with additional details as soon as it is available.

2. postmortem Jun 19, 2026, 01:52 PM UTC

   ## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers. QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally. ## Customer impact Impact was limited to the EU region and to two effects within the window above: * **Delivery delays / failures** _\(09:11 – 09:45 UTC\)_ — messages dispatched through affected instances may have failed on their first attempt. Thanks to automatic retries with backoff, most were re-delivered once connectivity recovered; messages that exhausted their retry policy during the window followed their configured failure path \(e.g., DLQ / failure callback\). * **Allowlist rejections** _\(09:11 – 11:24 UTC\)_ — customers who restrict inbound traffic to QStash's outbound IP ranges may have seen requests from the affected servers rejected. Customers who do not enforce source-IP allowlisting were unaffected by this. ## Root cause A routine networking configuration change contained an error that: 1. disrupted outbound connectivity on the affected nodes, and 2. caused affected nodes to acquire outbound IPs outside the advertised range. The underlying gap was systemic: the change procedure had no automated validation gate to catch the faulty state before it reached production. ## What we've done * **Fail-fast pre-checks** — automated validation now halts the change procedure _before it runs_ if a configuration would degrade connectivity or violate expected state. * **Egress IP-range enforcement** — outbound IP assignments are now checked against the advertised QStash range, so a node can no longer come online with an unadvertised IP. * **Tighter post-change verification** — completion now confirms outbound reachability and egress-IP conformance across affected nodes. ## Customer action None required. All QStash traffic again originates from the advertised outbound IP ranges. We apologize for the disruption.

Read the full incident report →

Started Jun 04, 2026, 10:10 AM UTC· resolved Jun 04, 2026, 02:09 PM UTC

1. investigating Jun 04, 2026, 10:10 AM UTC

   We are currently experiencing a service disruption affecting Upstash Vector in the IAD region. Initial investigation indicates that the issue is related to a Fly.io networking/routing problem impacting connectivity in the region. The Fly.io team is actively investigating the underlying cause. We will continue to monitor the situation and provide updates as more information becomes available.

2. monitoring Jun 04, 2026, 12:56 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 04, 2026, 02:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 29, 2026, 08:18 PM UTC· resolved May 30, 2026, 07:51 AM UTC

1. investigating May 29, 2026, 08:18 PM UTC

   Some Upstash Vector indexes in the US-EAST-1 region may be experiencing intermittent slowness. Initial investigation indicates this may be related to memory pressure on the affected servers. Our team is actively investigating the root cause and working to restore normal performance. We will provide updates as soon as more information is available.

2. monitoring May 29, 2026, 11:44 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 30, 2026, 07:51 AM UTC

   The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing memory utilization on the impacted nodes, rebalancing affected workloads where needed, and increasing available headroom capacity across the region. All affected indexes should now be operating normally, and based on the mitigations applied, we do not expect this issue to recur. We apologize for any inconvenience this may have caused.

Read the full incident report →

Started May 22, 2026, 11:27 PM UTC· resolved May 23, 2026, 12:00 AM UTC

1. investigating May 22, 2026, 11:27 PM UTC

   New database provisioning is currently failing due to an ongoing upstream provider issue affecting DNS. Existing databases and connections are unaffected — only the creation flow is impacted. We're monitoring the situation and will update once the upstream issue is resolved.

2. monitoring May 22, 2026, 11:48 PM UTC

   Upstream issue resolved — new database creation is working again. We're monitoring to confirm full recovery.

3. resolved May 23, 2026, 12:00 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 14, 2026, 03:47 PM UTC· resolved May 14, 2026, 05:31 PM UTC

1. monitoring May 14, 2026, 03:47 PM UTC

   We identified intermittent connection issues affecting some regions. Fix is being deployed. We'll update the list of affected regions shortly.

2. identified May 14, 2026, 03:49 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 14, 2026, 04:00 PM UTC

   A fix has been implemented and we are monitoring.

4. resolved May 14, 2026, 05:31 PM UTC

   Earlier today, unexpected load on our proxies caused intermittent connection issues for Upstash Redis in the following regions: us-east-1, us-west-1, ap-southeast-2, and ap-south-1. During this period, some clients may have seen connection timeouts or elevated error rates when reaching their databases. Our team identified the issue quickly and applied workarounds to relieve pressure on the affected proxies. Connection health has since been restored and we've been monitoring the regions to confirm everything is stable. All systems are now operating normally. We appreciate your patience and apologize for any disruption this may have caused.

Read the full incident report →

Started May 12, 2026, 09:49 AM UTC· resolved May 12, 2026, 01:48 PM UTC

1. investigating May 12, 2026, 09:49 AM UTC

   Some regions are experiencing connectivity issues due to an ongoing network problem. We are currently investigating

2. identified May 12, 2026, 10:50 AM UTC

   The issue has been identified and the fix is being implemented.

3. monitoring May 12, 2026, 12:05 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 12, 2026, 01:48 PM UTC

   This incident has been resolved.

5. postmortem May 15, 2026, 12:43 PM UTC

   On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.

Read the full incident report →

Started May 11, 2026, 03:05 PM UTC· resolved May 11, 2026, 06:22 PM UTC

1. investigating May 11, 2026, 03:05 PM UTC

   Some databases may experience increased latency or timeouts in Fly.io’s FRA region.

2. investigating May 11, 2026, 03:07 PM UTC

   We are continuing to investigate the issue.

3. investigating May 11, 2026, 05:19 PM UTC

   We are working with Fly team to investigate the root cause.

4. resolved May 11, 2026, 06:22 PM UTC

   The incident has been resolved. We are working with Fly team on RCA.

5. postmortem May 15, 2026, 12:49 PM UTC

   On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruption.

Read the full incident report →

Started May 08, 2026, 09:46 AM UTC· resolved May 08, 2026, 10:08 AM UTC

1. investigating May 08, 2026, 09:46 AM UTC

   We are currently investigating the issue.

2. investigating May 08, 2026, 09:46 AM UTC

   We are continuing to investigate this issue.

3. monitoring May 08, 2026, 10:06 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 08, 2026, 10:08 AM UTC

   This incident has been resolved, we will publish RCA soon.

5. postmortem May 12, 2026, 08:25 AM UTC

   **Root Cause Analysis** On **April 24**, we deployed a more optimized scheduler implementation in the **US East \(N. Virginia\)** region. On **May 8**, a user who had active schedules deleted their account. Under normal behavior, scheduled tasks associated with a deleted account should wake up, detect that the account no longer exists, and exit after performing cleanup. Due to a bug introduced in the new scheduler implementation, this code path did not return early as intended. Execution continued and resulted in a nil pointer dereference. A second issue then amplified the impact. When a panic occurs in the scheduler, it is designed to be recovered, logged, and isolated so that the process remains healthy. Because of another bug in the panic recovery path, the panic was not properly caught, which caused the worker process handling the scheduled job to terminate. After that process exited, another worker picked up responsibility for delivering the same scheduled task. Since the same faulty execution path was still present, that worker also failed. This created a cascading failure pattern across workers attempting to process the affected schedules. **Resolution** We deployed two fixes: * Added the missing early return in the deleted-account cleanup path, preventing the nil pointer dereference. * Corrected the panic recovery logic so that future panics are safely recovered, logged, and reported without causing worker processes to terminate. With these changes in place, the affected execution path is now safe. Even if a future bug triggers a panic in this area, it will be isolated and reported rather than causing process-level failure.

Read the full incident report →

Started May 05, 2026, 02:24 PM UTC· resolved May 06, 2026, 08:05 AM UTC

1. identified May 05, 2026, 02:24 PM UTC

   We are currently experiencing issues in the US region. - Duplicate Deliveries: During this period, some scheduled jobs may be executed twice. - Schedule Disruption: Schedules created between April 24, 2026 and May 2, 2026 are currently not running. Our team is actively working on a fix. Once the migration is complete, affected schedules will resume normal operation. We will provide updates as progress continues.

2. identified May 05, 2026, 02:29 PM UTC

   We are continuing to work on a fix for this issue.

3. monitoring May 05, 2026, 03:44 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring May 05, 2026, 09:31 PM UTC

   Main schedule functionality is back to normal. We are currently checking if previously created schedules are delivered as expected before marking the incident as resolved.

5. resolved May 06, 2026, 08:05 AM UTC

   This incident has been resolved.

6. postmortem May 06, 2026, 08:55 AM UTC

   # **Incident Postmortem: Scheduled Jobs Inconsistency in US Region** On May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update. The issue has been resolved, and all affected schedules have been restored. ## **Summary** As part of an ongoing scalability improvement, we recently updated scheduling infrastructure in the US region to a new architecture. During this transition, a legacy execution path remained in the codebase as a fallback mechanism. On May 1, a bug caused the system to revert to the legacy path. This resulted in inconsistent state between the old and new scheduling systems for some users. ## **Impact** The incident affected a limited number of users in the US region. **Most users were not affected**, and the vast majority of schedules continued operating normally throughout the incident. Users who did not update schedules during the transition window continued operating normally throughout the incident. **A subset of users who created, edited, paused, or deleted schedules between April 24 and May 1 may have experienced one or more of the following:** * Schedule updates not being reflected * Paused schedules becoming active again * Deleted schedules reappearing * Newly created schedules not executing as expected Schedules created after the transition may have stopped executing briefly before recovery. ## **Root Cause** During the transition, the new scheduling infrastructure became the source of truth for schedule state. Due to a bug, the system unexpectedly reverted traffic to the legacy scheduling path, which began accepting updates independently from the new system. This caused the two systems to diverge and resulted in inconsistent schedule state for affected users. ## **Resolution** After identifying the issue, we: 1. Restored the new scheduling system as the active source of truth 2. Reconciled data between the legacy and new systems 3. Updated missing schedule changes back into the new infrastructure 4. Performed conflict resolution to preserve user data and schedule continuity In some cases, schedules that had previously been paused or deleted were restored to avoid permanent data loss. ## **Preventive Measures** We are implementing several changes to prevent similar incidents: * Removing obsolete fallback execution paths after transitions complete * Adding automated safeguards and alerts for unexpected system fallback behavior * Improving consistency validation between systems * Expanding rollback and reconciliation testing We apologize for the disruption and appreciate everyone’s patience while we resolved the issue.

Read the full incident report →

Started Apr 27, 2026, 04:09 PM UTC· resolved Apr 27, 2026, 07:13 PM UTC

1. identified Apr 27, 2026, 04:09 PM UTC

   Some users on QStash US might be experiencing schedule issues.

2. monitoring Apr 27, 2026, 07:04 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 27, 2026, 07:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 02, 2026, 06:45 PM UTC· resolved Apr 02, 2026, 07:53 PM UTC

1. identified Apr 02, 2026, 06:45 PM UTC

   Servers in the iad region experienced unexpected disk load, resulting in elevated latencies and a temporary read-only state. We are migrating replicas to new instances to mitigate the issue and expect to have it fully resolved shortly.

2. resolved Apr 02, 2026, 07:53 PM UTC

   Replication complete, incident resolved.

Read the full incident report →

Started Mar 27, 2026, 03:30 PM UTC· resolved Mar 27, 2026, 03:30 PM UTC

1. resolved Mar 27, 2026, 05:05 PM UTC

   Due to a race condition in a process that attaches static IPs to nodes, some of the IPs in the dns were detached from the nodes, causing timeouts.

Read the full incident report →

Started Mar 06, 2026, 01:50 PM UTC· resolved Mar 06, 2026, 02:49 PM UTC

1. investigating Mar 06, 2026, 01:50 PM UTC

   We are currently investigating this issue.

2. monitoring Mar 06, 2026, 02:08 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 06, 2026, 02:49 PM UTC

   This incident has been resolved.

4. postmortem Mar 09, 2026, 10:07 AM UTC

   On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on a subset of nodes. Our team mitigated the incident by scaling up capacity in the region and redistributing load across additional nodes. Service recovered once the additional capacity was brought online. Resolution We have increased the number of machines in the Tokyo region to provide additional headroom and reduce the likelihood of similar incidents during traffic spikes. Next Steps We are continuing to review capacity safeguards and connection-handling limits to improve resilience against sudden traffic surges.

Read the full incident report →

Started Jan 23, 2026, 03:30 PM UTC· resolved Jan 23, 2026, 03:35 PM UTC

1. resolved Jan 23, 2026, 05:05 PM UTC

   We identified the cause of elevated latency impacting some databases in us-east-1 region between 15:30–15:35 UTC as a sudden surge of connection attempts that hit OS-level connection limits on our proxy layer. This resulted in slower new connection establishment and increased latency for some requests. Databases were not impacted. We are implementing additional proxy-level metrics and safeguards to detect and manage similar edge cases earlier.

Read the full incident report →

Started Dec 10, 2025, 11:50 AM UTC· resolved Dec 10, 2025, 12:08 PM UTC

1. investigating Dec 10, 2025, 11:50 AM UTC

   We identified an issue in Regional Databases in US-East-1 where database replicase have connectivity issues with each other. Other regions are not impacted. Global databases are not impacted. We are working on the issue.

2. resolved Dec 10, 2025, 12:08 PM UTC

   Issue has been identified and replicas were successfully reconnected.

Read the full incident report →

Started Dec 05, 2025, 09:03 AM UTC· resolved Dec 05, 2025, 09:18 AM UTC

1. investigating Dec 05, 2025, 09:03 AM UTC

   Upstream provider confirmed an incident. We are investigating the impact and potential resolutions.

2. resolved Dec 05, 2025, 09:18 AM UTC

   This incident has been resolved.

Read the full incident report →