Heron Data experienced a major incident on March 1, 2023 affecting Root and Transactions and 1 more component, lasting 1h 6m. The incident has been resolved; the full update timeline is below.
Affected components
RootTransactionsDashboard
Update timeline
- investigating Mar 01, 2023, 09:47 AM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- monitoring Mar 01, 2023, 10:53 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 01, 2023, 10:54 AM UTC
This incident has been resolved.
- postmortem Mar 01, 2023, 10:54 AM UTC
We apologise for the increased 500 errors and timeouts experienced in the last 10 hours. We introduced a new index to our database which was partially built and resulted in an “invalid” state. This resulted in long running queries which blocked subsequent reads. We dropped the index, as well as blocking database processes, and are now recovered.