Happo experienced a critical incident on June 7, 2021 affecting API and Web UI, lasting 3h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 07, 2021, 09:36 AM UTC
We are currently investigating downtime on our whole happo.io cluster.
- identified Jun 07, 2021, 10:35 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jun 07, 2021, 10:51 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jun 07, 2021, 10:59 AM UTC
All services are slowly starting to get back to normal. One part of the fix involved updating DNS entries for the happo.io domain. These updates can take a little while to propagate through different networks, so things will continue to be a little bit bumpy for another hour or so.
- monitoring Jun 07, 2021, 11:54 AM UTC
DNS updates are still propagating, but the service is back up and running in a majority of areas. We're still seeing some workers struggling (especially ios-safari and ipad-safari) but we're hoping these will be back up in the next hour as well.
- resolved Jun 07, 2021, 12:20 PM UTC
As far as our monitoring can detect, this issue is now fully resolved. Reach out to [email protected] if you're still affected by this!
- postmortem Jun 07, 2021, 02:58 PM UTC
This incident was caused by a combination of changes in our Kubernetes cluster. On Friday, we were working on routing certain requests to [happo.io](http://happo.io) through a caching layer \(to improve image loading performance\). That work never finished and left our loadbalancer in an unfortunate state where it couldn’t update its configuration. At this point though, the service was working fine \(and it kept working well over the weekend\). On Monday, we deployed a new version of the main [happo.io](http://happo.io) server. This update caused the load balancer routing to fail, because it was holding on to references to old services, that were now being killed. It took us a while to figure this out, and one of our attempts to get the service back up involved restarting the load balancer. This however, led to the whole cluster getting a new IP address with no way of recovering the old IP address. An update to our DNS \(pointing [happo.io](http://happo.io) to the new load balancer IP address\) made things work again, but it took a few hours before the DNS update had propagated to different parts of the world. Happo is often a required step for many organizations and the fact that it wasn’t available for a few hours definitely caused disruption to many users. We’re truly sorry this happened and we’ll do our best to prevent similar scenarios in the future. If you have questions, don’t hesitate to reach out directly to me at [[email protected]](mailto:[email protected]) – Henric Trotzig