Wasabi Outage History

On 10 January 2024 from 15:57 UTC to 16:45 UTC, we experienced an issue where clients received 4XX level errors while accessing the S3 buckets in the eu-south-1 \(Milan\) region. During an internal configuration change, a software bug caused the invalidation of TLS certificates used by internal servers to establish mutual TLS handshakes with load balancers. Consequently, the load balancers could not successfully connect to the internal servers, leading to 4XX errors returned to client requests. Wasabi’s Engineering and Operations teams identified the issue and manually reconfigured the TLS certificates to restore functionality. A fix was deployed to the eu-south-1 region to address the underlying bug and to prevent this problem from happening again.

On 06 January 2025 from 05:54 UTC to 08:02 UTC, Wasabi Account Control Manager Console \(WACM\) was unavailable, resulting in HTTP 4XX or 5XX errors when trying to access the WACM URL. The issue was caused by a failure in restarting internal services responsible for maintaining web service availability. This failure was further worsened by delays due to heavy database load, which the service relies on. Once the team identified a database lock as the cause of the slowness and the extended time required to reestablish the service, the long-running query causing the delay was terminated. The services were then restarted successfully, restoring the web service to its normal state.

On Sunday 17 November at approximately 00:14 UTC, an incident occurred in the eu-central-2 \(Paris\) region, resulting in elevated HTTP 5XX error responses and slow response times to buckets impacting some customers. During an upgrade to the region, an error caused the service to allocate more connections than needed resulting in an overload to one of the service components. This caused an increase in errors returned to a subset of customer buckets which were served by this component. To address the situation, the Operations Team reverted the update to restore service. By 15:11 UTC on 17 November 2024 service in the eu-central-2 region was fully restored to normal operational status. The error was corrected and the update was subsequently re-deployed successfully.

From 2024-10-31 at 21:10 UTC to 2024-11-01 at 04:00 UTC, we experienced issues with a subset of customer Buckets that were unavailable due to an unexpected power distribution issue while a planned power maintenance window was in progress at the EU-Central-1 region. The maintenance window required to run some servers on a single source and when the switch occurred, the power infrastructure could not keep up with the power draw from some of the infrastructure racks resulting in power down for those racks. The impact resulted in failures in Reads/Write operations and Billing for impacted buckets. Our Operations team worked with the Equinix Datacenter team to restore the power and service availability was re-established as soon as the maintenance was over, and power was completely restored.

From 2024-10-31 00:42 UTC to 2024-10-31 10:03 UTC, we experienced an issue where Control Accounts, Sub-Accounts and Invoice data in WACM was not loading properly when attempted to be viewed by the client. Upon investigation by our Engineering Team, it was found that the nodes responsible for fetching and delivering this data were in a bad state. Once the nodes were restored to a functional state, our team started the process of re-indexing the data, which would allow the data to be served via the WACM portal successfully. At 10:03 UTC on October 31, 2024, the WACM data was fully available to customers.

From 2024-09-18 17:55 UTC to 2024-09-18 18:08 UTC, we experienced an issue where client connection attempts to Wasabi Cloud Storage were impacted during an upgrade to our services in the CA-CENTRAL-1 region. The activity had no expectations to create any service impact and began on 2024-09-18 at 13:00 UTC. During the maintenance task, two internal software components had a networking port conflict, causing a delay on the restart process for our load balancer network component. Once our Operations Team identified the condition and cause for the failure, immediate corrective action was taken, and service availability was reestablished.

From 2024-09-16 04:30 UTC to 2024-09-16 12:30 UTC, we experienced an issue within our US-EAST-1 region causing customers to receive 5XX errors and a reduced ability to ingest data to customer buckets within the region. The system user-servers reached capacity with logs resulting in a failure of our streaming service. Because the user-servers were busy writing logs, they had reduced capability to handle requests. Additionally, messages that were unable to be published were written to disk, further increasing I/O operations on the system. By 12:30 UTC, our Operations team had taken corrective action by emptying the streaming queue, and restarted the user-server services. After these actions were performed, ingest to our US-EAST-1 region had been fully restored.

From 2024-06-10 18:25 UTC to 2024-06-10 18:43 UTC, we experienced an issue in our US-EAST-1 \(Ashburn\) vault which resulted in degraded performance in the region. This impacted customer’s ability to send and retrieve data to/from their buckets in the region, and access services such as the Wasabi Web Console and Wasabi Account Control API. At around 17:48 UTC, the Wasabi Operations team began deploying updates in the US-EAST-1 region. After the update was completed and the process of restarting internal services was started, one service failed, causing communication between our client servers and database to fail. Our Operations Team took action by removing the applied configuration of the failed service and redeploying the service. After this action, the internal services came up successfully and communication between our client servers and database was restored, restoring service in the region. At 18:43 UTC all services in the US-EAST-1 region were operational.