Wasabi experienced a minor incident on January 10, 2025 affecting EU-South-1 (Milan), lasting 1h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 10, 2025, 04:30 PM UTC
We are currently investigating issues in our EU-SOUTH-1 (Milan) region.
- monitoring Jan 10, 2025, 04:51 PM UTC
We have identified and corrected the issue, and our EU-SOUTH-1 (Milan) region is now back to a fully operational state. We will continue to monitor the region.
- resolved Jan 10, 2025, 06:03 PM UTC
This incident has been resolved.
- postmortem Jan 15, 2025, 08:01 PM UTC
On 10 January 2024 from 15:57 UTC to 16:45 UTC, we experienced an issue where clients received 4XX level errors while accessing the S3 buckets in the eu-south-1 \(Milan\) region. During an internal configuration change, a software bug caused the invalidation of TLS certificates used by internal servers to establish mutual TLS handshakes with load balancers. Consequently, the load balancers could not successfully connect to the internal servers, leading to 4XX errors returned to client requests. Wasabi’s Engineering and Operations teams identified the issue and manually reconfigured the TLS certificates to restore functionality. A fix was deployed to the eu-south-1 region to address the underlying bug and to prevent this problem from happening again.