Transcend incident

Degraded performance of our Backend API [EU Region]

Minor Resolved View vendor source →

Transcend experienced a minor incident on April 25, 2025 affecting Backend API - EU, lasting 13h 2m. The incident has been resolved; the full update timeline is below.

Started
Apr 25, 2025, 05:56 AM UTC
Resolved
Apr 25, 2025, 06:59 PM UTC
Duration
13h 2m
Detected by Pingoru
Apr 25, 2025, 05:56 AM UTC

Affected components

Backend API - EU

Update timeline

  1. investigating Apr 25, 2025, 05:56 AM UTC

    We are currently investigating performance issues with our backend API.

  2. identified Apr 25, 2025, 01:37 PM UTC

    We have identified a fix and are scaling our infrastructure to accommodate extreme API demand. The first-order impact to load times in the Admin Dashboard were fixed overnight and we are monitoring results this morning (Pacific Time). The second-order impact on DSR processing times is our focus this morning, and infrastructure scaling is underway.

  3. monitoring Apr 25, 2025, 04:23 PM UTC

    We have successfully scaled our infrastructure up to accommodate the increased API demand in the EU region. Backend performance is now healthy. However, the secondary effect of the recent backend performance degradation is that a backlog of DSRs have accrued. The backlog is actively being processed now. We will keep this incident open until that DSR processing pipeline returns to normal levels.

  4. resolved Apr 25, 2025, 06:59 PM UTC

    The DSR backlog is now down to normal levels. Incoming DSRs should process as per usual.