Neo4j Aura incident

GDS functions missing from AuraDS Enterprise and Professional instances

Major Resolved View vendor source →

Neo4j Aura experienced a major incident on January 10, 2025 affecting AuraDS on AWS (*.databases.neo4j.io) and AuraDS Enterprise on AWS (*.databases.neo4j.io) and 1 more component, lasting 2d 17h. The incident has been resolved; the full update timeline is below.

Started
Jan 10, 2025, 07:49 PM UTC
Resolved
Jan 13, 2025, 01:41 PM UTC
Duration
2d 17h
Detected by Pingoru
Jan 10, 2025, 07:49 PM UTC

Affected components

AuraDS on AWS (*.databases.neo4j.io)AuraDS Enterprise on AWS (*.databases.neo4j.io)AuraDS Enterprise on Azure (*.databases.neo4j.io)AuraDS on Azure (*.databases.neo4j.io)AuraDS Enterprise on GCP (*.databases.neo4j.io)AuraDS on GCP (*.databases.neo4j.io)

Update timeline

  1. investigating Jan 10, 2025, 07:49 PM UTC

    We are investigating an issue where GDS functions are missing from AuraDS Enterprise and AuraDS instances.

  2. identified Jan 10, 2025, 10:02 PM UTC

    We have identified the issue causing the GDS plugin to be missing from the impacted instances and are working towards restoring the plugin to all missing instances.

  3. identified Jan 11, 2025, 12:45 AM UTC

    We have identified the issue causing the GDS plugin to be missing from the impacted instances and we are working on releasing the fix.

  4. monitoring Jan 11, 2025, 03:54 AM UTC

    We have identified the issue causing the GDS plugin to be missing from the impacted instances and have rolled out a fix. If your GDS plugins are missing from your instance, please pause and resume your instance as a workaround until all remaining impacted instances are remediated.

  5. resolved Jan 13, 2025, 01:41 PM UTC

    We have monitored the service and this incident is now resolved.

  6. postmortem Jan 31, 2025, 02:37 PM UTC

    ## **What happened** On January 10, 2025, following a system update, some AuraDS instances experienced an issue where Graph Data Science \(GDS\) functionality became temporarily unavailable resulting in users receiving “unknown function” error messages while attempting to use GDS functions. Users with AuraDS Self-serve and AuraDS Enterprise instances were affected. ### **Timeline:** 2025-01-10 15:39:00 - Code change merged and rolled out to production 2025-01-10 17:26:59 - Incident started - Customers begin reporting GDS function unavailability 2025-01-10 19:49:00 - Status Page updated with incident once initial scope determined 2025-01-11 00:09:00 - Fix merged and deployed to production. Begin process of restoring plugins to missing instances 2025-01-14 10:45:15 - Incident resolved - Fix fully deployed and verified ## **Root Cause** Neo4j rolled out an update which affects how GDS plugin assignments are managed for AuraDS instances. Incident-1966 unexpectedly resulted in the removal of the GDS library from AuraDS Self-Serve and AuraDS Enterprise instances. ## **Customer** **Impact** * AuraDS instances lost access to GDS functionalities * A number of projections were lost due to the remediation activities to restore GDS functionality, affecting a number of user instances * A number of users needed to restart their GDS workflows, once a fix was put in place ## **Resolution** Neo4j worked to identify the root cause, affected instances, and deploy a fix to restore the GDS functionality as the first step. After step one, Neo4j updated affected instances to load the GDS plugins and verify the fix across all impacted customers’ instances. ## **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Improve on our Console reconciler testing methodologies * Increased observability metrics intended to detect and fix plugin-related issues more quickly * Improve deployment procedures by adding more effective guardrails against unintended plugin changes