Cloudera incident
EU Control plane has issues accessing workload UIs
Cloudera experienced a major incident on November 3, 2023 affecting Cloudera Management Console, lasting 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 03, 2023, 10:28 AM UTC
We are currently investigating a potential issue with EU Control Plane down(workload logins/UI are not working). We will have an update within 60 mins. Customer Experience: Customers may observe issues while logging in to the atlas URL, Ranger URL, and resource manager URL are not accessible.
- investigating Nov 03, 2023, 10:32 AM UTC
We are continuing to investigate this issue.
- resolved Nov 03, 2023, 11:29 AM UTC
Our teams have successfully deployed a fix for the issue and confirmed that the issue has been resolved. If you are still experiencing issues or have any questions please raise a support case with us. A root cause analysis (RCA) will be published within seven business days. Incident Start Time: 09:40 AM UTC November 3rd, 2023 Incident End Time: 11:00 AM UTC November 3rd, 2023
- postmortem Nov 10, 2023, 01:48 PM UTC
On Nov 3rd 2023 at ~09:40 UTC, few customers of the Europe Control plane reported issues about Workloads User Interface not being accessible via Management Console. Please do note the workload clusters were operational during this time. Upon further investigation it was identified that a recent production change hit an edge case exposing a new bug. The same was fixed immediately by rolling a new change. This issues did not manifest in our lower environments, where the changes were tested prior to rolling out to production. As a mitigation action, our teams are working on adding additional test cases to improve the coverage for this corner case.