Some nodes may be down for short time due to database issues

Update timeline

1. identified Jan 15, 2024, 09:31 AM UTC

   Brief downtime expected. Our team has identified database issues that require immediate attention to ensure the continued reliability and performance of our services. No action required from the customers.

2. identified Jan 15, 2024, 09:53 AM UTC

   Issue: Database Overload Current Status: We are currently undergoing operational changes to bring all nodes back online. Certain nodes are experiencing downtime . Operational Changes Details: In an effort to improve database performance and scalability, we are implementing critical operational changes. The changes are designed to optimize the infrastructure and enhance overall system reliability Next Steps: Following the completion of the changes, nodes will be brought back online progressively to minimize any service impact. Impact: SOME NODES DOWN: Temporary unavailability of services on affected nodes during the operational changes. Users may experience disruptions until the changes are successfully implemented. Contact: For urgent inquiries or additional information, please contact our support team. Next update is expected at 12:00 CET We apologize for any inconvenience caused and thank you for your patience and cooperation.

3. identified Jan 15, 2024, 11:16 AM UTC

   The solution is still under investigation. Next update is expected at 14:00 CET

4. identified Jan 15, 2024, 01:57 PM UTC

   Database recovery is in progress. We are bringing nodes up in a controlled way. Next update is expected at 16:00 CET

5. identified Jan 15, 2024, 04:30 PM UTC

   The majority of the nodes is back to active. We are bringing the remaining nodes up.

6. monitoring Jan 15, 2024, 07:22 PM UTC

   All nodes are stable now - we'll be monitoring until tomorrow

7. resolved Jan 16, 2024, 10:06 AM UTC

   Resolved: Database Incident Current Status: All nodes are now up and running smoothly. The database incident has been successfully resolved. Resolution Details: Our cloud team has diligently addressed the root cause of the database outage. Critical measures have been taken to ensure the stability and performance of the entire database infrastructure. Next Steps: Continuous monitoring and proactive measures will be in place to maintain optimal database performance. A comprehensive post-incident report will be prepared and shared with our users shortly. We appreciate your patience and understanding during the resolution of this incident. If you experience any lingering issues or have concerns, please reach out to our support team. Thank you for your cooperation. The Exalate team

8. postmortem Apr 01, 2026, 04:13 AM UTC

   timeout