Wasabi incident
Wasabi Account Control Manager Console views for Control Accounts, Sub-Accounts and Invoices not loading
Wasabi experienced a minor incident on October 30, 2024 affecting Wasabi Account Control Manager Console, lasting 13h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2024, 11:14 PM UTC
We are currently investigating this issue and expect to have these views restored shortly.
- investigating Oct 31, 2024, 12:34 AM UTC
While some of the views have been restored, we continue to work to resolve the issue. This page will be updated as more progress is made.
- identified Oct 31, 2024, 12:44 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Oct 31, 2024, 09:55 AM UTC
This incident has been resolved.
- postmortem Nov 07, 2024, 03:24 PM UTC
From 2024-10-31 00:42 UTC to 2024-10-31 10:03 UTC, we experienced an issue where Control Accounts, Sub-Accounts and Invoice data in WACM was not loading properly when attempted to be viewed by the client. Upon investigation by our Engineering Team, it was found that the nodes responsible for fetching and delivering this data were in a bad state. Once the nodes were restored to a functional state, our team started the process of re-indexing the data, which would allow the data to be served via the WACM portal successfully. At 10:03 UTC on October 31, 2024, the WACM data was fully available to customers.