Wasabi experienced a minor incident on May 24, 2025 affecting US-East-2 (N. Virginia), lasting 5h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 24, 2025, 11:22 AM UTC
We are currently experiencing system errors in our US-EAST-2 region. Customers may experience elevated HTTP 5XX error responses when interacting with their Wasabi bucket(s). We will update this page as we have more information.
- identified May 24, 2025, 12:46 PM UTC
The issue has been identified, and we are making appropriate changes to the system.
- monitoring May 24, 2025, 01:10 PM UTC
We have made changes, traffic is operating normally, and we are monitoring the systems.
- resolved May 24, 2025, 04:29 PM UTC
This incident has been resolved. Please contact [email protected] if you continue to see errors related to this incident.
- postmortem Jun 03, 2025, 03:44 AM UTC
On 23 May 2025 at 23:00 UTC, Wasabi’s us-east-2 \(Manassas\) vault began seeing a large increase in S3 API traffic, causing regional resources to become strained with excessive internal connections. This excessive, sustained increase in traffic to the region was outside of normal operating conditions. At 08:12 UTC on 24 May 2025, this high resource usage began to impact the vault, causing an increase in 5xx errors for API calls to S3 buckets. Upon investigation from our Operations Team, it was found that a small number of accounts were responsible for consuming a large portion of available resources within the region, causing the increase in HTTP 5XX error responses for other customers. Our Operations Team took action by limiting the available connections to these accounts, which allowed for the overall active connections to the region to drop substantially. By 08:32 UTC on 24 May 2025, the region was back to normal operation.