Happeo experienced a minor incident on March 26, 2024 affecting Web Application, lasting 1d 9h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 26, 2024, 07:42 AM UTC
We are seeing difficulties accessing channels & posts. We are investigating the issue.
- identified Mar 26, 2024, 08:54 AM UTC
We have identified the issue and working towards resolving the ongoing problems.
- monitoring Mar 26, 2024, 10:45 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Mar 26, 2024, 03:28 PM UTC
Although Happeo is mostly operating as normal, we are still seeing some very minor degradation in our systems. We will continue to monitor and make small tweaks where needed until all services return to normal.
- monitoring Mar 27, 2024, 11:05 AM UTC
Starting 26.03.2024 at around 02:30 CET we have detected a bug in the Google infrastructure we use to perform data replication within the microservices that power Happeo different features. During the first hours of the incident, Happeo became unavailable at times, a situation that was controlled by our Engineering team by providing extra capacity to our cloud resources. Despite Happeo being functional and available, we are still slightly affected by the incident and we perceive a minor degradation in data replication, which may cause some minor usability issues to Happeo's users (for example, not being able to create a post, or adding/removing channel members). We are actively working together with Google to fix the problem and we will get you updated as soon as we have updated information.
- resolved Mar 27, 2024, 05:26 PM UTC
We have resolved the issue originating from Google and all services should be running as expected again.