Litium experienced a major incident on March 11, 2025 affecting Cloud API and Database services and 1 more component, lasting 23h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 11, 2025, 10:58 AM UTC
We have detected issues with the App Services for LCC serverless. Investigation is ongoing. More information will be provided shortly.
- identified Mar 11, 2025, 11:48 AM UTC
We have identified an underlaying cause of the issue and working together with the supplier to mitigate.
- identified Mar 11, 2025, 12:20 PM UTC
We are currently experiencing issues due to underlying hardware problems at our supplier. Our team is actively collaborating with them to resolve the issue as quickly as possible. We appreciate your patience and understanding during this time.
- identified Mar 11, 2025, 01:04 PM UTC
Our supplier's technical team is currently on-site at the datacenter, working to replace and fix the faulty hardware causing the issues. We are closely monitoring the situation and will provide further updates as soon as possible. Thank you for your continued patience and understanding.
- identified Mar 11, 2025, 02:32 PM UTC
Some systems are already back online, and our team continue to work diligently to restore normal operations on the remainder.
- identified Mar 11, 2025, 04:19 PM UTC
Additional systems are now online. The team continue to work on restoring the rest.
- identified Mar 11, 2025, 05:37 PM UTC
We are continuing our efforts to restore all systems to full operational status. Our team is exploring multiple approaches to ensure a swift resolution. We appreciate your patience and will keep you informed of any further developments.
- identified Mar 11, 2025, 06:55 PM UTC
We are pleased to inform you that another key service has been restored, and most customers should now be operational. However, there are still some services that we are actively working on. Our team remains dedicated to resolving all issues as quickly as possible. Thank you for your continued patience and support.
- identified Mar 11, 2025, 08:24 PM UTC
Most services are now up and running. We are still working on resolving issues with our Litium Search service. Our team is dedicated to restoring full functionality as quickly as possible. Thank you for your patience and understanding.
- monitoring Mar 11, 2025, 10:33 PM UTC
All services are now up and running. We will continue to monitor the situation closely until tomorrow morning to ensure stability. Following this, we will initiate our incident review process to thoroughly analyze the cause and impact of the disruption. Thank you for your patience and understanding.
- resolved Mar 12, 2025, 10:47 AM UTC
We have monitored the situation closely overnight and can confirm that all services have remained functioning normally throughout the night. If anyone continues to experience issues, please contact support. The incident was caused by underlying hardware problems at our supplier and affected some of our cloud services. We will continue our analysis of the incident, create a detailed incident report, and take necessary measures to prevent this from happening again. We sincerely apologize for the inconvenience caused by the incident.