Increase experienced a major incident on March 26, 2024 affecting API, lasting 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 26, 2024, 04:09 PM UTC
We're looking into an issue that's leading to API unavailability, starting at 16:02 UTC.
- resolved Mar 26, 2024, 04:26 PM UTC
A fix has rolled out and the API is back in a healthy state as of 16:26 UTC.
- postmortem Mar 26, 2024, 07:24 PM UTC
24 minutes of API outage was caused by a failed database migration. The outage affected all Increase users. At 16:00 UTC, we executed a migration against Increase’s database. The migration partially failed, resulting in an inconsistency between the deployed code and database schema. We were immediately alerted to the failure. By 16:08 UTC, the scope of the issue had been confirmed and the engineering team gathered in a Zoom room to debug the issue and restore service. At 16:18 UTC, the engineering team performed a data migration and deployed a new version of the API server to be in sync with the database schema. Full API service was restored by 16:24 UTC. Increase is making changes to our Continuous Integration pipeline to ensure that database migrations of this type cannot result in data being out-of-sync with the code. Our alerting and escalation paths worked as expected. We apologize for the outage and appreciate your patience as we worked to restore service. For help with any specific transaction, please contact [[email protected]](mailto:[email protected]).