Wasabi incident
Degraded performance in the EU-West-2 (Paris) region
Wasabi experienced a minor incident on November 17, 2024 affecting EU-West-2 (Paris), lasting 2h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 17, 2024, 12:06 AM UTC
We are currently investigating degraded performance in the EU-West-2 (Paris) region
- investigating Nov 17, 2024, 12:48 AM UTC
Our Operations team continues to work with the network traffic team to isolate the problem, we will update this page with more information soon
- monitoring Nov 17, 2024, 01:21 AM UTC
We have identified the issue and are making changes to address the traffic. We are continuing to monitor the system as well
- resolved Nov 17, 2024, 02:14 AM UTC
This incident is now resolved.
- postmortem Dec 06, 2024, 11:32 AM UTC
On Sunday 17 November at approximately 00:14 UTC, an incident occurred in the eu-central-2 \(Paris\) region, resulting in elevated HTTP 5XX error responses and slow response times to buckets impacting some customers. During an upgrade to the region, an error caused the service to allocate more connections than needed resulting in an overload to one of the service components. This caused an increase in errors returned to a subset of customer buckets which were served by this component. To address the situation, the Operations Team reverted the update to restore service. By 15:11 UTC on 17 November 2024 service in the eu-central-2 region was fully restored to normal operational status. The error was corrected and the update was subsequently re-deployed successfully.