Microsoft Azure experienced a major incident on January 4, 2022, lasting 19h 11m. The incident has been resolved; the full update timeline is below.
Update timeline
- postmortem Jan 05, 2022, 07:41 AM UTC
Summary of Impact: Between 12:30 UTC on 04 Jan 2022 and 7:41 UTC on 5 Jan 2022, customers with Azure Cosmos DB accounts in East US may have experienced connectivity and service availability errors while accessing their Cosmos DB databases. One Cosmos DB cluster in the East US region was unavailable during this time, so both new and existing connections to databases in this subscription in this region may have resulted in errors or timeouts. Root Cause: Cosmos DB uses Azure Service Fabric as the underlying platform for providing fault tolerance in the cluster. Service Fabric uses the ring topology, and each node establishes a lease relationship with nodes in its proximity (i.e. neighborhood) to detect failure. It has a set of nodes that are responsible for determining cluster memberships of other nodes, known as Arbitrators. A node that fails to refresh lease within a timeout period will be reported by its neighbors, and the arbitrators need to determine whether the node should leave the cluster. This check is done in a timer callback. During this incident, the timer callback on one of the nodes was fired multiple times at a frequency higher than intended. This resulted in the node's neighbors getting incorrectly reported as unavailable. By design, the Arbitrators trusted this information as they did not receive any healthy uptime notification within the stipulated time frame. This continued until the quorum of nodes was lost, and the cluster went down eventually. The cluster came back up once the culprit node was manually rebooted as part of the mitigation efforts. Mitigation: After the initial investigation, the cluster was marked as offline at 14:08 UTC on 04 Jan 2022 which triggered regional failover for accounts that had multiple regions and automatic failover enabled. Customers that did not have automatic failover enabled continued to be impacted until the cluster was recovered. The cluster was recovered by rebooting the Service Fabric Infrastructure nodes after removing the culprit node. However, recovery of the cluster was delayed due to overload of the configuration store as the service was restarting. Cosmos DB Engineers initially tried to reduce the load on the configuration store by delaying the startup of about 20% of the nodes. This approach did not fully resolve the problem. Engineers then manually applied configuration changes to increase the timeout on the requests used to fetch data from the configuration store. This change allowed the recovery to continually make progress. Availability to the cluster was incrementally restored as service back end processes started running. Recovery was completed at 07:41 UTC on 05 Dec 2022. Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): Service Fabric team to develop a fix to improve resilience in case of misfired timer(s) reporting incorrect node health status within Azure Service Fabric. Azure Cosmos DB to improve monitoring to better identify culprit nodes early on if this failure pattern reoccurs. Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey