Happo Outage History
Happo is up right nowPingoru has recorded 28 incidents for Happo: 4 in the last 12 months and 1 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 0 of 720 (0%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 1 | 3 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 513 (0%) |
Incidents, newest first
Major August 17, 2026 API failures 2h 38m API
Started Aug 17, 2026, 02:54 PM UTC· resolved Aug 17, 2026, 05:33 PM UTC
- investigating Aug 17, 2026, 02:54 PM UTC
We’re seeing errors coming from our own API calls to GitHub causing most jobs to fail on our end as well. We’re investigating this right now.
- resolved Aug 17, 2026, 06:33 PM UTC
A GitHub outage caused some of our API endpoints started to fail. To reduce the impact we briefly turned off some communication with GitHub which made things stabilize on our side. At this time, no Happo statuses could be posted to GitHub PRs and commits. At 18.11 UTC we saw things starting to stabilize and briefly after we were back in a good state again.
Major April 22, 2026 Timeouts and failed API requests 7m
Started Apr 22, 2026, 04:30 PM UTC· resolved Apr 22, 2026, 04:37 PM UTC
- resolved Apr 22, 2026, 05:01 PM UTC
During a window of about 7 minutes, our API and web application served at happo.io was unresponsive. This was caused by a configuration mistake in our backend systems.
Major April 21, 2026 GitHub statuses not working 16m API, Web UI
Started Apr 21, 2026, 11:53 AM UTC· resolved Apr 21, 2026, 12:10 PM UTC
- identified Apr 21, 2026, 11:53 AM UTC
We are investigating an issue with Happo not being able to post build statuses to PRs at the moment.
- resolved Apr 21, 2026, 12:10 PM UTC
This issue has now been fixed.
Critical April 18, 2026 Happo.io temporarily down –
Started Apr 18, 2026, 11:22 AM UTC· resolved Apr 18, 2026, 11:22 AM UTC
- resolved Apr 18, 2026, 11:26 AM UTC
While applying an infrastructure change to our cluster, our load balancer was misconfigured and ended up in a state where it sent HTTP traffic to the wrong nodes. The issue was discovered right away but it took a while to figure out the root cause (load balancer config). External traffic was briefly affected with API requests to happo.io timing out. The issue has since been resolved and all services are back up and running.
Major June 23, 2025 General downtime 43m API, Chrome, Web UI, Firefox and 4 more
Started Jun 23, 2025, 03:00 PM UTC· resolved Jun 23, 2025, 03:43 PM UTC
- resolved Jun 23, 2025, 03:59 PM UTC
We are continuing to monitor an issue which caused most Happo jobs and API calls to fail. We are seeing things recover right now and will keep an eye on this to make sure we're not regressing again. Issue started at 15:00 UTC and was ongoing until 15:43 UTC.
- postmortem Jun 23, 2025, 04:27 PM UTC
The root cause of the downtime and slowness from the API was a new database query that we deployed a few days ago. Once it had some traffic we started noticing slow queries coming from the new query. It took us a about an hour to track everything down, and we quickly reverted the code that added the new query. We will continue to monitor things but as of right now the system is stable.
Major June 2, 2025 Increased error rates for snap-request jobs 1h 12m API
Started Jun 02, 2025, 07:05 PM UTC· resolved Jun 02, 2025, 08:17 PM UTC
- investigating Jun 02, 2025, 07:05 PM UTC
We are investigating an issue with snap-requests randomly failing with a `HTTP error 404: http://happo-view-service/api/snap-requests/:id/payload -- Not Found` error.
- resolved Jun 02, 2025, 08:17 PM UTC
We have found the root cause of this issue now. A temporary fix is currently active and we are soon to deploy a proper fix. The issue was caused by a stale cache that caused snap-request jobs to not save properly.
Notice April 9, 2025 Service outage –
Started Apr 09, 2025, 03:10 PM UTC· resolved Apr 09, 2025, 03:10 PM UTC
- resolved Apr 09, 2025, 03:10 PM UTC
During approximately 20 minutes, happo.io and the Happo API was broken. A majority of API calls would fail with a 502 Bad Gateway error, and the website was down. We are still investigating the root cause, but for now we have all systems running again. Expect some delay while our workers pick up work from our backlog of pending jobs.