Happo had 4 outages in the last 2 years totaling 16m of downtime — averaging 0.2 incidents per month.
There were 4 Happo outages since June 23, 2025 totaling 16m of downtime. Each is summarised below — incident details, duration, and resolution information.
During a window of about 7 minutes, our API and web application served at happo.io was unresponsive. This was caused by a configuration mistake in our backend systems.
We are investigating an issue with Happo not being able to post build statuses to PRs at the moment.
This issue has now been fixed.
While applying an infrastructure change to our cluster, our load balancer was misconfigured and ended up in a state where it sent HTTP traffic to the wrong nodes. The issue was discovered right away but it took a while to figure out the root cause (load balancer config). External traffic was briefly affected with API requests to happo.io timing out. The issue has since been resolved and all services are back up and running.
We are continuing to monitor an issue which caused most Happo jobs and API calls to fail. We are seeing things recover right now and will keep an eye on this to make sure we're not regressing again. Issue started at 15:00 UTC and was ongoing until 15:43 UTC.
The root cause of the downtime and slowness from the API was a new database query that we deployed a few days ago. Once it had some traffic we started noticing slow queries coming from the new query. It took us a about an hour to track everything down, and we quickly reverted the code that added the new query. We will continue to monitor things but as of right now the system is stable.