Happo experienced a major incident on July 4, 2023 affecting API and Web UI, lasting 3h 9m. The incident has been resolved; the full update timeline is below.
Affected components
APIWeb UI
Update timeline
- investigating Jul 04, 2023, 02:40 PM UTC
We're currently looking into why happo.io is slower than usual to respond. We're seeing slow database queries and a number of failed jobs.
- identified Jul 04, 2023, 02:48 PM UTC
We're temporarily setting the service in maintenance mode to ease down on traffic to our database.
- monitoring Jul 04, 2023, 03:35 PM UTC
Things have settled down and we're seeing requests succeed again. We'll keep an eye on things to ensure the system continues to be stable.
- resolved Jul 04, 2023, 05:49 PM UTC
The system has been stable for the last two hours. We will continue monitoring things over the next couple of days, and follow up with fixes to the root cause.