Neo4j Aura experienced a critical incident on December 28, 2023 affecting Aura Console (console.neo4j.io) and Aura API (api.neo4j.io), lasting 6h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 28, 2023, 02:17 PM UTC
We are currently aware of an outage effecting both the Aura Console and Aura API. We are currently investigating and taking action to restore service.
- investigating Dec 28, 2023, 03:18 PM UTC
We are aware of an outage effecting both the Aura Console and Aura API. We are continuing to investigate and are taking action to restore service.
- investigating Dec 28, 2023, 04:20 PM UTC
We are aware of an outage continuing to effect both the Aura Console and Aura API. We are continuing to investigate and have ruled out some components, and are taking ongoing action to restore service.
- identified Dec 28, 2023, 05:02 PM UTC
We have identified an issue with a service component and are working to resolve that as quickly as possible to restore service to Aura Console and Aura API.
- monitoring Dec 28, 2023, 05:30 PM UTC
The component identified to be causing the outage has been remedied. Aura Console and Aura API are operational and we are currently monitoring the health of both.
- resolved Dec 28, 2023, 08:35 PM UTC
The incident impacting Aura Console and Aura API has been resolved, and stable for ~3 hours. We are considering this incident resolved, and systems are fully operational. We will update this incident with details regarding the outage after thorough investigation and review.
- postmortem Jan 16, 2024, 10:18 PM UTC
### **What happened** Customers started to report errors accessing the Console and the Aura API on 2023-12-28 at 13:50 UTC. An incident was raised and our engineering teams identified requests to GCP Datastore timing out as the cause of the unavailability. We were particularly affected due to our usage of the python Datastore drivers version selected. We raised a support ticket with our cloud service provider \(GCP\) and in the meantime our SREs identified the issue and mitigated it. Service availability was restored around 2023-12-28 at 17:30 UTC. ### **How the service was affected** Both the Aura Console and API make use of the GCP Datastore service for user management. Console access authentication was successful but loading the Aura tenant information was affected and blocked the display of the Aura Console UI. Aura API also operates at tenant level and was impacted. Requests started timing out causing the unavailability. ### **What we are doing now** We have taken steps to update our GCP Datastore’s driver version according to GCP’s recommendation as well as making sure we better handle an outage and timeout on some queries to prevent blocking. We will also implement a circuit breaker in our logic to reduce the impact of an outage. Finally we will be looking into improving our detection and alerting in case of a GCP Datastore service outage.