Upstash incident

QStash: Degraded performance

Minor Resolved View vendor source →

Upstash experienced a minor incident on April 18, 2025 affecting EU-CENTRAL-1, lasting 42m. The incident has been resolved; the full update timeline is below.

Started
Apr 18, 2025, 12:36 PM UTC
Resolved
Apr 18, 2025, 01:18 PM UTC
Duration
42m
Detected by Pingoru
Apr 18, 2025, 12:36 PM UTC

Affected components

EU-CENTRAL-1

Update timeline

  1. investigating Apr 18, 2025, 12:36 PM UTC

    We are currently investigating this issue.

  2. investigating Apr 18, 2025, 12:36 PM UTC

    We are continuing to investigate this issue.

  3. monitoring Apr 18, 2025, 12:46 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Apr 18, 2025, 01:18 PM UTC

    This incident has been resolved.

  5. postmortem Apr 18, 2025, 01:19 PM UTC

    An internal cleanup task for QStash events, coinciding with disc layer compaction task has caused performance degradation to some users. ‌ Mitigation: Team has mitigated the event by pausing some of these tasks and monitored the status for a while. ‌ Fixes: Improvements are being applied to these tasks to use resources more gracefully. Disk resources are increased to be able to handle a bigger burst of load.