Happo experienced a major incident on June 23, 2025 affecting API and Chrome and 1 more component, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Jun 23, 2025, 03:59 PM UTC
We are continuing to monitor an issue which caused most Happo jobs and API calls to fail. We are seeing things recover right now and will keep an eye on this to make sure we're not regressing again. Issue started at 15:00 UTC and was ongoing until 15:43 UTC.
- postmortem Jun 23, 2025, 04:27 PM UTC
The root cause of the downtime and slowness from the API was a new database query that we deployed a few days ago. Once it had some traffic we started noticing slow queries coming from the new query. It took us a about an hour to track everything down, and we quickly reverted the code that added the new query. We will continue to monitor things but as of right now the system is stable.