Litium incident
Resolved: Issues with Litium Search affecting Serverless Cloud workloads depending on Litium Search
Litium experienced a major incident on September 25, 2024 affecting Cloud API and Litium Search, lasting 4h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 25, 2024, 08:59 AM UTC
Some jobs in Serverless Cloud CLI is stuck in "pending" status. We are currently investigating this issue.
- monitoring Sep 25, 2024, 10:11 AM UTC
A fix has been implemented and we are monitoring the results.
- investigating Sep 25, 2024, 10:22 AM UTC
Issues with some sites in Serverless Cloud reported. We are currently investigating this issue.
- investigating Sep 25, 2024, 10:28 AM UTC
We are continuing to investigate this issue.
- identified Sep 25, 2024, 11:01 AM UTC
We have identified an issue affecting Litium Search. Our team is actively working on a solution and we are making progress toward implementing a fix. We will provide further updates as soon as possible. We appreciate your patience and understanding during this time.
- monitoring Sep 25, 2024, 11:25 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 25, 2024, 11:41 AM UTC
Since the fix has been rolled out majority of workload is working as normal, but some customers may still affected, we are continuing to monitor and resolving remaining issues.
- monitoring Sep 25, 2024, 12:14 PM UTC
Litium Search is now fully operational. We continuing to monitor for any further issues.
- monitoring Sep 25, 2024, 12:35 PM UTC
All services are now fully operational. Note that it may take some extra time for all jobs to process since jobs has queued up.
- resolved Sep 25, 2024, 01:49 PM UTC
This incident has been resolved.