Upstash Outage History

Partial outage due to http parsing errors. Problem was resolved couple minutes later. If problem continues, refreshing the DNS cache is recommended.

This incident has been resolved, we will publish RCA soon.

Between 01:18 and 04:03 UTC, customers were unable to access running boxes or perform create, read, update, or delete operations. Running containers were not affected and continued operating throughout. The issue has been fully resolved and all box operations are functioning normally. We are taking all necessary measures to prevent this from happening again. We apologize for the disruption.

This incident has been resolved.

The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing memory utilization on the impacted nodes, rebalancing affected workloads where needed, and increasing available headroom capacity across the region. All affected indexes should now be operating normally, and based on the mitigations applied, we do not expect this issue to recur. We apologize for any inconvenience this may have caused.

This incident has been resolved.

Earlier today, unexpected load on our proxies caused intermittent connection issues for Upstash Redis in the following regions: us-east-1, us-west-1, ap-southeast-2, and ap-south-1. During this period, some clients may have seen connection timeouts or elevated error rates when reaching their databases. Our team identified the issue quickly and applied workarounds to relieve pressure on the affected proxies. Connection health has since been restored and we've been monitoring the regions to confirm everything is stable. All systems are now operating normally. We appreciate your patience and apologize for any disruption this may have caused.

This incident has been resolved.

The incident has been resolved. We are working with Fly team on RCA.

This incident has been resolved.

Replication complete, incident resolved.

Due to a race condition in a process that attaches static IPs to nodes, some of the IPs in the dns were detached from the nodes, causing timeouts.

This incident has been resolved.

We identified the cause of elevated latency impacting some databases in us-east-1 region between 15:30–15:35 UTC as a sudden surge of connection attempts that hit OS-level connection limits on our proxy layer. This resulted in slower new connection establishment and increased latency for some requests. Databases were not impacted. We are implementing additional proxy-level metrics and safeguards to detect and manage similar edge cases earlier.

We identified and fixed a bug that could cause messages with Flow Control enabled to be delayed longer than their configured delay, resulting in unexpectedly long pending times. The fix is in place and the issue should not recur. If you’re still seeing unusually long-delayed messages, please contact [email protected] and we can help with remediation.

Issue has been identified and replicas were successfully reconnected.

This incident has been resolved.

A fix has been deployed as a workaround so that our systems are not affected from the ongoing incident of the cloud provider

As a side effect of an incident on the underlying cloud provider, Upstash Console has had availability issues between 07:00UTC and 09:23UTC. Only Upstash Console is impacted, Upstash products remained operational.

Between 15:46–15:55 UTC, some client connection attempts to databases in us-east-1 timed out due to unexpected high load on a server. The node was recovered at 15:50 UTC, and the updated DNS record propagated by 15:55 UTC. Services are operating normally.

Impact: A subset of clients connecting through the eu-central-1 region experienced increased error rates and timeouts when accessing certain databases. Clients in us-west-2 were also briefly affected. The issue was limited in scope and did not impact other regions. Root Cause: During an ongoing migration to improve database routing reliability, a configuration step was applied inconsistently across regions. Resolution: Our monitoring alerted us within minutes, and the migration was promptly rolled back for the affected regions. Service definitions were restored, and normal database connectivity resumed by 15:08 UTC. Next Steps: We are reviewing our migration process to ensure consistency across all regions and adding additional safeguards to prevent similar issues in the future.