Wasabi Outage History

Started Apr 08, 2025, 08:00 PM UTC· resolved Apr 08, 2025, 08:16 PM UTC

1. resolved Apr 11, 2025, 03:16 PM UTC

   On 08 April 2025 from 20:05 UTC to approximately 20:16 UTC, Wasabi services in our London (eu-west-3) region was unavailable due to an internal routing issue which impacted the region's ability to process incoming customer requests. At 20:05 UTC, our Operations Team was notified by our internal monitoring service that customer traffic was being impacted within the region. Upon investigation, it was found that internal DNS routing within the eu-west-3 region was failing to resolve internal hosts. Wasabi's Operations Team re-deployed the DNS stack and once re-deployment was completed at 20:16 UTC, customer connections to the region was restored.

Read the full incident report →

Started Apr 08, 2025, 02:41 PM UTC· resolved Apr 08, 2025, 03:23 PM UTC

1. investigating Apr 08, 2025, 02:41 PM UTC

   We are currently investigating system errors in our AP-NORTHEAST-2 region. Our team is working to isolate the issue and provide a resolution.

2. monitoring Apr 08, 2025, 02:45 PM UTC

   We have identified and resolved the issue. All affected resources are back in service. We are currently monitoring the status. If you see any error, please reach out to [email protected] for assistance.

3. resolved Apr 08, 2025, 03:23 PM UTC

   This incident has been resolved.

4. postmortem Apr 11, 2025, 03:05 PM UTC

   On 08 April 2025 from 14:13 UTC to approximately 14:39 UTC, Wasabi services in our Osaka \(ap-northeast-2\) region was unavailable due to an internal routing issue which impacted the region's ability to process incoming customer requests. At 14:13 UTC, our Operations Team was notified by our internal monitoring service that customer traffic was being impacted within the region. Upon investigation, it was found that internal DNS routing within the ap-northeast-2 region was failing to resolve internal hosts. Wasabi's Operations Team re-deployed the DNS stack and once re-deployment was completed at 14:39 UTC, customer connections to the region was restored.

Read the full incident report →

Started Apr 04, 2025, 10:01 AM UTC· resolved Apr 04, 2025, 01:59 PM UTC

1. investigating Apr 04, 2025, 10:01 AM UTC

   We are currently investigating system errors in our EU-CENTRAL-1 region. Our team is working to isolate the issue and provide a resolution.

2. investigating Apr 04, 2025, 10:05 AM UTC

   We are continuing to investigate this issue.

3. monitoring Apr 04, 2025, 12:34 PM UTC

   We have identified and resolved the issue. All affected resources are back in service. We are currently monitoring the status. If you see any error, please reach out to [email protected] for assistance.

4. resolved Apr 04, 2025, 01:59 PM UTC

   This incident has been resolved.

5. postmortem Apr 21, 2025, 03:01 PM UTC

   On 04 April 2025 from 06:20 UTC to 12:20 UTC, we experienced system errors in our eu-central-1 \(Amsterdam\) region. This issue affected the S3 service and impacted customer’s ability to read and write to/from their Wasabi buckets within the region. Starting at 05:11 UTC, our Operations Team was notified of a spike in internal system read operations in our eu-central-1 region across multiple storage zones. Approximately 1 hour and 9 minutes after this spike at 06:20 UTC, our storage sub-systems errored to read and write requests, impacting customers. Wasabi’s Operations Team was able to identify a recent upgrade to our internal system as responsible for the spike in storage read operations, that eventually led to the storage system errors. Operations proceeded to rollback this upgrade to a previous stable release, which then restored operations on the storage system, leading to a restoration of service for customers. By 12:20 UTC, all S3 services in the eu-central-1 region were operational.

Read the full incident report →

Started Mar 27, 2025, 10:09 AM UTC· resolved Mar 27, 2025, 10:14 AM UTC

1. resolved Mar 28, 2025, 01:53 AM UTC

   Between 10:09PM to 10:14PM UTC, we experienced errors with connecting to the Wasabi Web Console, STS, and IAM services. It was a temporary issue, auto-resolved and Operations Team validated it. There is no longer any further impact, and all services are now running without issue.

2. postmortem Apr 03, 2025, 09:10 PM UTC

   On 27 March 2025 from 10:09 UTC to approximately 10:14 UTC, Wasabi services in our Ashburn \(us-east-1\), Manassas \(us-east-2\), Hillsboro \(us-west-1\), Plano \(us-central-1\), Amsterdam \(eu-central-1\), Tokyo \(ap-northeast-1\), Osaka \(ap-northeast-2\), London \(eu-west-1\), and Frankfurt \(eu-central-2\) regions were unavailable due to an issue, impacting customer traffic and Web Console connectivity. Wasabi Engineering and Operations teams were notified by our automated monitoring systems of an issue with a spike in connection between internal service worker nodes. This spike led to the inability to open new connections within the services. Wasabi’s Operations Team manually restarted the services, which corrected the issue and allowed for client connections to resume successfully. By 10:14 UTC, all internal services were restarted and all services returned to normal operating conditions.

Read the full incident report →

Started Mar 17, 2025, 10:34 AM UTC· resolved Mar 17, 2025, 10:50 AM UTC

1. resolved Mar 17, 2025, 05:53 PM UTC

   Between 10:40 to 10:50 UTC, we experienced errors with connecting to the Wasabi Web Console, STS, and IAM services. Our Operations Team has taken corrective action and there is no longer any further impact, and all services are now running without issue.

2. postmortem Mar 31, 2025, 04:02 AM UTC

   On 17 March 2025 from 10:34 UTC to approximately 10:50 UTC, Wasabi services in our Ashburn \(us-east-1\), Manassas \(us-east-2\), Hillsboro \(us-west-1\), Plano \(us-central-1\), Amsterdam \(eu-central-1\), Tokyo \(ap-northeast-1\), Osaka \(ap-northeast-2\), London \(eu-west-1\), and Frankfurt \(eu-central-2\) regions were unavailable due to an issue which impacted the ability for internal services to communicate effectively within all Wasabi services. Wasabi Engineering and Operations teams were notified by our automated monitoring systems at 10:34 UTC of an issue with a spike in connection between internal service worker nodes. This spike led to the inability to open new connections within the services. Wasabi’s Operations Team noted that automatic restarts of the services did not clear the issue, prompting manual intervention to perform a manual restart of internal service nodes, which corrected the issue and allowed for client connections to resume successfully. By 10:50 UTC, all internal services were restarted and all services returned to normal operating conditions.

Read the full incident report →

Started Feb 19, 2025, 09:00 AM UTC· resolved Feb 19, 2025, 11:00 AM UTC

1. resolved Feb 20, 2025, 09:25 PM UTC

   From 09:00 to 11:00 UTC on 19 February 2025, visitors experienced HTTP 5XX errors when attempting to view our Wasabi.com website. These errors were caused by an update to the site, which resulted in multiple modules not loading properly. Once the issue was identified, the changes were reverted, and the website was fully operational around 11:00 UTC.

Read the full incident report →

Started Feb 05, 2025, 05:35 AM UTC· resolved Feb 05, 2025, 06:55 AM UTC

1. investigating Feb 05, 2025, 05:59 AM UTC

   We are currently investigating an issue to access the Wasabi Account Control Manager Console page. Our customers might experience 404 and 502 errors when accessing the Wasabi Account Control Manager Console.

2. identified Feb 05, 2025, 06:07 AM UTC

   Our teams had identified the issue and are working on the fix.

3. monitoring Feb 05, 2025, 06:09 AM UTC

   The action to fix for the issue (errors 404 and 502) to access the Wasabi Account Control Manager Console was completed and the service has returned to normal. We are currently monitoring it.

4. resolved Feb 05, 2025, 06:55 AM UTC

   This incident has been resolved.

5. postmortem Feb 10, 2025, 10:37 PM UTC

   On 05 February 2025 from 05:06 AM UTC to approximately 06:04 AM UTC, Wasabi Account Control Manager Console \(WACM\) was unavailable, resulting in errors 404 or 502 when trying to access the WACM URL. The issue was caused by internal component restart with a scenario of DB lock causing the slowness due to a backup routine and as consequence it took additional time for the service to be reestablished. Once the internal team stopped the backup routine, it was possible to proceed with the tasks needed and the restart action completed as expected, bringing the service up and running normally.

Read the full incident report →

Started Jan 10, 2025, 04:30 PM UTC· resolved Jan 10, 2025, 06:03 PM UTC

1. investigating Jan 10, 2025, 04:30 PM UTC

   We are currently investigating issues in our EU-SOUTH-1 (Milan) region.

2. monitoring Jan 10, 2025, 04:51 PM UTC

   We have identified and corrected the issue, and our EU-SOUTH-1 (Milan) region is now back to a fully operational state. We will continue to monitor the region.

3. resolved Jan 10, 2025, 06:03 PM UTC

   This incident has been resolved.

4. postmortem Jan 15, 2025, 08:01 PM UTC

   On 10 January 2024 from 15:57 UTC to 16:45 UTC, we experienced an issue where clients received 4XX level errors while accessing the S3 buckets in the eu-south-1 \(Milan\) region. During an internal configuration change, a software bug caused the invalidation of TLS certificates used by internal servers to establish mutual TLS handshakes with load balancers. Consequently, the load balancers could not successfully connect to the internal servers, leading to 4XX errors returned to client requests. Wasabi’s Engineering and Operations teams identified the issue and manually reconfigured the TLS certificates to restore functionality. A fix was deployed to the eu-south-1 region to address the underlying bug and to prevent this problem from happening again.

Read the full incident report →

Started Jan 06, 2025, 06:30 AM UTC· resolved Jan 06, 2025, 08:00 AM UTC

1. investigating Jan 08, 2025, 09:30 PM UTC

   We are currently investigating issues with accessing the WACM Console. Users may experience HTTP 404 or 502 error responses.

2. resolved Jan 08, 2025, 09:30 PM UTC

   This incident has been resolved.

3. postmortem Jan 08, 2025, 09:31 PM UTC

   On 06 January 2025 from 05:54 UTC to 08:02 UTC, Wasabi Account Control Manager Console \(WACM\) was unavailable, resulting in HTTP 4XX or 5XX errors when trying to access the WACM URL. The issue was caused by a failure in restarting internal services responsible for maintaining web service availability. This failure was further worsened by delays due to heavy database load, which the service relies on. Once the team identified a database lock as the cause of the slowness and the extended time required to reestablish the service, the long-running query causing the delay was terminated. The services were then restarted successfully, restoring the web service to its normal state.

Read the full incident report →

Started Dec 07, 2024, 04:14 PM UTC· resolved Dec 08, 2024, 12:28 AM UTC

1. investigating Dec 07, 2024, 04:14 PM UTC

   We are currently investigating an increase in 500 level HTTP responses on customer traffic to the us-east-1 and us-east-2 regions.

2. investigating Dec 07, 2024, 05:14 PM UTC

   We are continuing to investigate the system errors in the us-east-1 and us-east-2 regions. We will update this page as we have more information.

3. identified Dec 07, 2024, 06:25 PM UTC

   We have identified a power issue at the data center which is in the process of being restored. We will update this page as we have more information.

4. identified Dec 07, 2024, 07:24 PM UTC

   We are continuing the process of restoring power and bringing systems back up. We will update this page as we have more information.

5. identified Dec 07, 2024, 07:59 PM UTC

   Power has been fully restored to the us-east-2 region. We are continuing the process of restoration to the us-east-1 region and will update this page as we have more information.

6. identified Dec 07, 2024, 09:29 PM UTC

   Power has been fully restored to the us-east-1 region and we are continuing to work on bringing all systems back online. We will update this page as we have more information.

7. monitoring Dec 07, 2024, 10:17 PM UTC

   All systems are now back online and fully operational. We are continuing to monitor the regions and will update this page as we have more information.

8. resolved Dec 08, 2024, 12:28 AM UTC

   Services in both regions have been restored. Please reach out to [email protected] if you see any issues related to this incident.

9. postmortem Dec 16, 2024, 04:47 PM UTC

   On 7 December 2024 from 15:22 UTC to 21:30 UTC, Wasabi experienced a loss of power event in our US-EAST-1 and US-EAST-2 data centers. At 15:22, our Operations Team noticed that infrastructure within the US-EAST-1 and US-EAST-2 regions failed to respond to standard smoke tests and monitoring tools, and reviewing the activity for the regions indicated a full loss of power to all server racks and infrastructure within the building. At 16:00 UTC, Wasabi received confirmation from Iron Mountain that power loss for the entire building has occurred. By 16:15 UTC, Iron Mountain Operations begins to restore power to an incremental number of racks for Wasabi’s infrastructure, allowing our Operations Team to run systematic health checks across all server nodes, and by 21:30 UTC we have confirmation that all systems are running optimally and both US-EAST-1 and US-EAST-2 regions were fully operational.

Read the full incident report →

Started Dec 05, 2024, 09:11 PM UTC· resolved Dec 05, 2024, 11:49 PM UTC

1. investigating Dec 05, 2024, 09:11 PM UTC

   We are currently investigating an increase in "Account Connection Limited" responses from the S3 system on a number of Wasabi Storage Regions. We will update this page as we have more information.

2. monitoring Dec 05, 2024, 10:39 PM UTC

   We have isolated the issue "Account Connection Limited" errors and engineering has made changes to restore the service. We will update this page with any further information.

3. resolved Dec 05, 2024, 11:49 PM UTC

   After monitoring, we have confirmed the issue is resolved and there is no further impact on our services.

Read the full incident report →

Started Dec 02, 2024, 07:30 PM UTC· resolved Dec 03, 2024, 03:11 PM UTC

1. resolved Dec 03, 2024, 04:59 PM UTC

   From 19:30 UTC 12/2/24 to 15:11 UTC 12/3/24 customers may have experienced elevated HTTP 5XX error responses and slow response times to buckets in our eu-central-2 (Frankfurt) region. Retries to the region were successful and the region is now fully operational.

Read the full incident report →

Started Dec 01, 2024, 11:30 PM UTC· resolved Dec 01, 2024, 11:41 PM UTC

1. resolved Dec 01, 2024, 11:57 PM UTC

   We experienced an issue that interrupted services for a brief period at 23:41 UTC on December 1, 2024. The system config was adjusted and service resumed at approximately 23:51 UTC.

Read the full incident report →

Started Nov 17, 2024, 12:06 AM UTC· resolved Nov 17, 2024, 02:14 AM UTC

1. investigating Nov 17, 2024, 12:06 AM UTC

   We are currently investigating degraded performance in the EU-West-2 (Paris) region

2. investigating Nov 17, 2024, 12:48 AM UTC

   Our Operations team continues to work with the network traffic team to isolate the problem, we will update this page with more information soon

3. monitoring Nov 17, 2024, 01:21 AM UTC

   We have identified the issue and are making changes to address the traffic. We are continuing to monitor the system as well

4. resolved Nov 17, 2024, 02:14 AM UTC

   This incident is now resolved.

5. postmortem Dec 06, 2024, 11:32 AM UTC

   On Sunday 17 November at approximately 00:14 UTC, an incident occurred in the eu-central-2 \(Paris\) region, resulting in elevated HTTP 5XX error responses and slow response times to buckets impacting some customers. During an upgrade to the region, an error caused the service to allocate more connections than needed resulting in an overload to one of the service components. This caused an increase in errors returned to a subset of customer buckets which were served by this component. To address the situation, the Operations Team reverted the update to restore service. By 15:11 UTC on 17 November 2024 service in the eu-central-2 region was fully restored to normal operational status. The error was corrected and the update was subsequently re-deployed successfully.

Read the full incident report →

Started Nov 15, 2024, 01:21 PM UTC· resolved Nov 15, 2024, 05:16 PM UTC

1. investigating Nov 15, 2024, 01:21 PM UTC

   We are currently investigating intermittent issues affecting Wasabi Managment Console login.

2. monitoring Nov 15, 2024, 04:04 PM UTC

   We have identified and resolved the issue. The access to our Management Console is back to normal and operational again. We will continue to monitor our services.

3. resolved Nov 15, 2024, 05:16 PM UTC

   This issue was resolved by 1:55PM UTC.

Read the full incident report →

Started Oct 31, 2024, 09:44 PM UTC· resolved Nov 01, 2024, 06:49 AM UTC

1. investigating Oct 31, 2024, 09:44 PM UTC

   We have reports of power issues in our EU-CENTRAL-1 region and some users may be experiencing errors when interfacing with their Wasabi buckets hosted there. We will update our status page with more information as it becomes available.

2. investigating Oct 31, 2024, 10:21 PM UTC

   A facility power distribution issue is impacting several services in the eu-central-1 region. Direct Connect links are impacted as well as a subset of buckets. We continue to work with the facility to restore service.

3. identified Oct 31, 2024, 11:15 PM UTC

   We have identified the issue and impacted resources. Team continue to work to restore service.

4. monitoring Nov 01, 2024, 05:17 AM UTC

   The power distribution issue has been resolved and all affected resources are back in service. We are currently monitoring the status. If you see any error, please reach out to [email protected] for assistance.

5. resolved Nov 01, 2024, 06:49 AM UTC

   The incident has been resolved. Please reach out to [email protected] if you see any issues related to this incident.

6. postmortem Nov 08, 2024, 01:24 AM UTC

   From 2024-10-31 at 21:10 UTC to 2024-11-01 at 04:00 UTC, we experienced issues with a subset of customer Buckets that were unavailable due to an unexpected power distribution issue while a planned power maintenance window was in progress at the EU-Central-1 region. The maintenance window required to run some servers on a single source and when the switch occurred, the power infrastructure could not keep up with the power draw from some of the infrastructure racks resulting in power down for those racks. The impact resulted in failures in Reads/Write operations and Billing for impacted buckets. Our Operations team worked with the Equinix Datacenter team to restore the power and service availability was re-established as soon as the maintenance was over, and power was completely restored.

Read the full incident report →

Started Oct 30, 2024, 08:14 PM UTC· resolved Oct 31, 2024, 09:55 AM UTC

1. investigating Oct 30, 2024, 11:14 PM UTC

   We are currently investigating this issue and expect to have these views restored shortly.

2. investigating Oct 31, 2024, 12:34 AM UTC

   While some of the views have been restored, we continue to work to resolve the issue. This page will be updated as more progress is made.

3. identified Oct 31, 2024, 12:44 AM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Oct 31, 2024, 09:55 AM UTC

   This incident has been resolved.

5. postmortem Nov 07, 2024, 03:24 PM UTC

   From 2024-10-31 00:42 UTC to 2024-10-31 10:03 UTC, we experienced an issue where Control Accounts, Sub-Accounts and Invoice data in WACM was not loading properly when attempted to be viewed by the client. Upon investigation by our Engineering Team, it was found that the nodes responsible for fetching and delivering this data were in a bad state. Once the nodes were restored to a functional state, our team started the process of re-indexing the data, which would allow the data to be served via the WACM portal successfully. At 10:03 UTC on October 31, 2024, the WACM data was fully available to customers.

Read the full incident report →