Wasabi incident
Degraded performance in US-EAST-1 and US-EAST-2 regions
Wasabi experienced a minor incident on May 30, 2024 affecting US-East-1 (N. Virginia) and US-East-2 (N. Virginia) and 1 more component, lasting 4d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 30, 2024, 03:58 PM UTC
Wasabi's Operations Team has been informed that Iron Mountain Datacenters in the us-east-1 and us-east-2 regions are experiencing a cooling issue, impacting the operating temperatures of Wasabi server hardware in those regions.
- identified May 30, 2024, 04:24 PM UTC
We are seeing a large amount of HTTP 500-level errors being returned to client requests due to this incident. Please check the status page regularly to receive the latest updates.
- identified May 30, 2024, 07:29 PM UTC
Recovery operation is currently underway to bring back the impacted systems. This will take between 6 to 12 hours to complete. We will continue to update here as progress is made.
- identified May 31, 2024, 01:09 AM UTC
We are continuing the restoration of the impacted components. A large number of servers have been rebooted & restored and we are working on the remaining servers in both regions. We anticipate the full process to complete between 4-8 hours. We will continue to update our status page.
- monitoring May 31, 2024, 03:40 AM UTC
Systems have been restored to operational status. We continue to monitor these services. If you experience any issue please reach out to our support team.
- monitoring May 31, 2024, 09:15 PM UTC
Services to our us-east-1 and us-east-2 regions have been restored. Out of an abundance of caution we will continue to monitor the regions throughout the weekend. For any issues, please reach out to our Support Team at [email protected]
- resolved Jun 03, 2024, 06:53 PM UTC
Services in our us-east-1 and us-east-2 regions have been restored and all faults related to the data center cooling issues have been mitigated. For any issues, please reach out to our Support Team at [email protected]
- postmortem Jun 05, 2024, 08:00 PM UTC
From 2024-05-30 15:36 UTC to 2024-05-31 05:00 UTC, we experienced elevated temperatures in our us-east-1 and us-east-2 regions. This issue was due to a cooling system failure in one of the Iron Mountain Datacenter \(IMDC\) buildings which hosts our storage sub-system and database servers. This failure caused temperatures to pass a safe operating threshold, causing systems to involuntarily shutdown in order to prevent any damage. Between 2024-05-30 17:00:00 UTC and 2024-05-31 05:00:00 UTC, our Operations Team worked on each server rack to bring each individual component back up safely, ran integrity checks on the hardware and replaced faulty equipment. At 2024-05-31 05:00:00 UTC, our services were returned to a fully operational status.