Flightcontrol incident

Upstash unavailability in FRA region

Resolved after 18 hours 39 minutes, affecting Upstash for Redis, on September 23, 2026.

resolved Sep 24, 2026, 08:12 AM UTC

**Impact** From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. **What happened** During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. **What we're changing** • We will be adding tooling to return a replica to service safely after maintenance. • We will be adding safety checks to our upgrade process. • We will make the upgrade procedure more resilient to errors like this one.

Started
Sep 23, 2026, 12:42 PM UTC
Resolved
Sep 24, 2026, 07:22 AM UTC
Duration
18h 39m
Detected by Pingoru
Sep 23, 2026, 12:42 PM UTC

Affected components

Upstash for Redis

Update timeline

  1. investigating Sep 23, 2026, 12:42 PM UTC

    We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.

  2. investigating Sep 23, 2026, 12:42 PM UTC

    We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.

  3. monitoring Sep 23, 2026, 01:08 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Sep 23, 2026, 01:08 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Sep 23, 2026, 02:03 PM UTC

    This incident has been resolved.

  6. resolved Sep 23, 2026, 02:03 PM UTC

    This incident has been resolved.

  7. resolved Sep 24, 2026, 07:22 AM UTC

    Impact From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. What happened During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. What we're changing • We will be adding tooling to return a replica to service safely after maintenance. • We will be adding safety checks to our upgrade process. • We will make the upgrade procedure more resilient to errors like this one.

  8. resolved Sep 24, 2026, 07:22 AM UTC

    Impact From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. What happened During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. What we're changing • We will be adding tooling to return a replica to service safely after maintenance. • We will be adding safety checks to our upgrade process. • We will make the upgrade procedure more resilient to errors like this one.

  9. postmortem Sep 24, 2026, 07:22 AM UTC

    ## Impact From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. ## What happened During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. ## What we're changing * We will be adding tooling to return a replica to service safely after maintenance. * We will be adding safety checks to our upgrade process. * We will make the upgrade procedure more resilient to errors like this one.

  10. resolved Sep 24, 2026, 08:12 AM UTC

    **Impact** From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. **What happened** During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. **What we're changing** • We will be adding tooling to return a replica to service safely after maintenance. • We will be adding safety checks to our upgrade process. • We will make the upgrade procedure more resilient to errors like this one.

Your product runs on more than Flightcontrol

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →