Happo experienced a major incident on November 14, 2022 affecting Chrome and Firefox and 1 more component, lasting 2h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 14, 2022, 10:55 AM UTC
We are seeing elevated errors related to storing screenshots. We're investigating and hope to have an update soon!
- resolved Nov 14, 2022, 01:17 PM UTC
This incident has been resolved. The cause of the issue was a deploy we made last night (updating Safari, Chrome and Firefox to the latest versions). When saving a screenshot, we first check to see if the same image has already been stored. If so, we can avoid writing it to disk. There's a cache involved here when checking to see if the image was already present. This cache was partially broken, causing an error to be thrown under certain circumstances. A fix was made right away and we've rolled it out to all of our infrastructure.