Wasabi incident

Issues in US-EAST-1

Minor Resolved View vendor source →

Wasabi experienced a minor incident on May 13, 2024, lasting —. The incident has been resolved; the full update timeline is below.

Started
May 13, 2024, 01:03 PM UTC
Resolved
May 13, 2024, 12:00 AM UTC
Duration
Detected by Pingoru
May 13, 2024, 01:03 PM UTC

Update timeline

  1. resolved May 13, 2024, 01:03 PM UTC

    Between 23:56 to 00:22 UTC we saw issues with accepting incoming client requests in our US-EAST-1 region. Our Operations Team has resolved this issue and the region is now operating normally.

  2. postmortem May 17, 2024, 06:23 PM UTC

    On 12 May 2024 at 23:56 UTC until 13 May 2024 at 00:22 UTC, Wasabi had an issue in the US-EAST-1 \(Ashburn\) vault which resulted in degraded performance in the region. This impacted customer’s ability to send and retrieve data to/from their buckets in the region. The queueing service in US-EAST-1 region failed to accept new connections due to an overloaded memory issue. This caused issues with communication to our global database cache, which resulted in an inability to serve customer requests. Our Operations Team restarted the global database cache and queuing service. The restarting of both services resulted in connections being restored for the vault. At 00:22 UTC, 13 May 2024, all services were restored to the US-EAST-1 region.