Wasabi incident

System Errors in AP-NORTHEAST-1

Minor Resolved View vendor source →

Wasabi experienced a minor incident on May 21, 2025 affecting AP-Northeast-1 (Tokyo), lasting 3h 20m. The incident has been resolved; the full update timeline is below.

Started
May 21, 2025, 09:47 AM UTC
Resolved
May 21, 2025, 01:07 PM UTC
Duration
3h 20m
Detected by Pingoru
May 21, 2025, 09:47 AM UTC

Affected components

AP-Northeast-1 (Tokyo)

Update timeline

  1. investigating May 21, 2025, 09:47 AM UTC

    We are currently experiencing system errors in our AP-NORTHEAST-1 region. Customers may experience elevated HTTP 5XX error responses when interacting with their Wasabi bucket(s)/objects. We will update this page as we have more information.

  2. identified May 21, 2025, 10:42 AM UTC

    The issue has been identified and we are working towards to a solution.

  3. monitoring May 21, 2025, 11:13 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved May 21, 2025, 01:07 PM UTC

    This incident has been resolved, please contact [email protected] with any outstanding questions.

  5. postmortem May 30, 2025, 01:21 PM UTC

    On 21 May 2025 at 02:45 UTC, Wasabi’s Tokyo vault was impacted by system errors, causing client requests to result in 5xx errors for any API calls to their S3 buckets. This issue was due to an internal failure in the Tokyo Region where the backend server had a hardware fault, impacting the cluster of which it was a member. Our Operations Team identified the failure and took corrective action by performing a cluster failover to resolve the issue, which returned the service back to an operational state at 10:55 UTC 21 May 2025.