Heron Data experienced a critical incident on April 6, 2023 affecting Root and Transactions and 1 more component, lasting 14h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 06, 2023, 07:16 PM UTC
We are currently investigating an issue where we have reached the maximum number of IDs for a table in our database. We have to schedule some emergency downtime to resolve the issue
- identified Apr 06, 2023, 07:17 PM UTC
The issue has been identified and a fix is being implemented.
- identified Apr 06, 2023, 07:40 PM UTC
We are continuing to work on a fix for this issue.
- identified Apr 06, 2023, 07:42 PM UTC
We are continuing to work on a fix for this issue.
- identified Apr 06, 2023, 09:17 PM UTC
We have remediated part of the issue. Now the outage is limited to async processing and any route that involves fetching transaction categories (e.g., delete & get transactions, end user endpoints like /summary, /profit_and_loss, and various reports). We are executing a fix which is underway for the remainder
- identified Apr 06, 2023, 10:55 PM UTC
We are running a backfill on the table in question which we believe will take ~6 hours, so we will provide another update around then
- identified Apr 07, 2023, 05:36 AM UTC
The database migration is complete and async processing is back online
- monitoring Apr 07, 2023, 06:04 AM UTC
A fix has been implemented and we are monitoring the results.
- identified Apr 07, 2023, 06:16 AM UTC
We have identified that a related database table is impacted, and are implementing a fix
- monitoring Apr 07, 2023, 10:52 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Apr 07, 2023, 10:59 AM UTC
We are continuing to monitor for any further issues.
- resolved Apr 07, 2023, 11:33 AM UTC
This incident has been resolved.