Neo4j Aura incident

AuraDB Pro, AuraDS Pro and Aura BC instances on GCP may experience creation delays

Minor Resolved View vendor source →

Neo4j Aura experienced a minor incident on March 13, 2025 affecting AuraDB Professional on GCP (*.databases.neo4j.io) and AuraDS on GCP (*.databases.neo4j.io) and 1 more component, lasting 13h 13m. The incident has been resolved; the full update timeline is below.

Started
Mar 13, 2025, 09:27 PM UTC
Resolved
Mar 14, 2025, 10:41 AM UTC
Duration
13h 13m
Detected by Pingoru
Mar 13, 2025, 09:27 PM UTC

Affected components

AuraDB Professional on GCP (*.databases.neo4j.io)AuraDS on GCP (*.databases.neo4j.io)AuraDB Business Critical (*.databases.neo4j.io) on GCP

Update timeline

  1. identified Mar 13, 2025, 09:27 PM UTC

    We have identified an issue in GCP causing some new AuraDB Pro, AuraDS Pro and Aura BC instances to get stuck in creation for an extended period of time. Updating instances in this cohort may also experience delay. We are already preparing a fix.

  2. identified Mar 14, 2025, 12:59 AM UTC

    we are continuing work on a fix for this issue.

  3. identified Mar 14, 2025, 12:59 AM UTC

    We are in the process of releasing a fix for this issue and would update with a status when it has been rolled out across all impacted tiers

  4. monitoring Mar 14, 2025, 06:48 AM UTC

    The fix for this issue has been successfully rolled out across all impacted tiers. We are currently monitoring the situation to ensure stability.

  5. resolved Mar 14, 2025, 10:41 AM UTC

    We have confirmed that the fix has addressed issues and all operations fully recovered.

  6. postmortem Apr 14, 2025, 03:48 PM UTC

    ### What happened On March 13, 2025, some of our customers experienced delays in the creation of AuraDB Pro, AuraDS Pro, and Aura BC databases on our platform. This was due to an unexpected issue with our database management system, which caused some databases to become stuck during the creation process. Our team quickly identified the problem and worked diligently to resolve it. By March 14, all affected databases were successfully created and operational. We are committed to ensuring the reliability of our services and have taken steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused and appreciate your understanding and patience. ### Root Cause Recently, we implemented a new security feature to enhance the way we manage access to our database systems. This involved setting up additional security permissions in the IAM Policy for each of our paid database services. While we were careful to stay within known limits for these permissions, we encountered an unexpected issue due to an undocumented restriction on the size of these permissions. This caused a problem where our system couldn't create new permissions for databases in certain areas, leading to delays in processing and updates. ### Customer Impact * AuraDB Pro, AuraDS Pro, and Aura BC Databases experienced issues where databases would get stuck during creation. * A total of 8 AuraDB Pro databases were directly affected by this issue. ### Resolution We promptly addressed the issue by reversing recent IAM changes to ensure the stability of our services. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Modifying the access management method to eliminate the possibility of hitting IAM limits * Implementing additional automated checks to prevent service disruptions * Improving our service resilience to handle similar scenarios