FeedbackFruits experienced a major incident on August 30, 2022 affecting API and API and 1 more component, lasting 8h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 30, 2022, 08:49 AM UTC
We are currently investigating partial unavailability of nodes in our EU cluster.
- identified Aug 30, 2022, 08:59 AM UTC
Our infrastructure provider Microsoft Azure is reporting a global issue with DNS errors on the version of Ubuntu that we are running. While they investigate the issue, we are considering potential steps to mitigate the disruption. See https://azure.status.microsoft for more information.
- identified Aug 30, 2022, 09:00 AM UTC
We are continuing to work on a fix for this issue.
- identified Aug 30, 2022, 09:26 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Aug 30, 2022, 09:50 AM UTC
All nodes are available again, but we keep monitoring the situation.
- resolved Aug 30, 2022, 05:07 PM UTC
This incident has been resolved.
- postmortem Aug 31, 2022, 02:38 PM UTC
FeedbackFruits works with Microsoft Azure to provide a hosting cluster per region to deliver our services. This cluster is built with redundancy and high availability in mind: multiple nodes provide capacity to schedule workloads on. At the start of the incident on August 30th, 2022, at 10:23, node availability \(eg: capacity\) in our EU cluster had gradually dropped to a critically low level, as a security software update being automatically rolled out over 24h had an unexpected side effects: making nodes unavailable without being able to recover themselves. More on this underlying issue, which affected a significant portion of Microsoft Azure Kubernetes services, in the screenshot below from the Azure status portal. This lack of capacity in our cluster led to increased error rates/failed requests, with some end users not being able to view learning activities in FeedbackFruits. After diagnosing the underlying issue, we started applying a manual mitigation to circumvent the faulty security software update and bring back capacity. Error levels started reducing around 11:00, with traffic to FeedbackFruits stabilizing fully at 11:14. Due to the nature of the underlying issue, we monitored our clusters more intensively for the next hours, to be able to prevent reoccurence while the root issue was being resolved by our cloud vendor. We’re awaiting Microsoft’s full post-mortem on this issue to assess whether we need to make changes to our configuration to prevent issues like these in the future. The faulty security update has since been corrected. We apologize for any inconvience caused. 