Products Up experienced a major incident on December 5, 2024 affecting Data Processing, lasting 10h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 05, 2024, 08:53 AM UTC
We are currently experiencing ongoing issues with jobs getting stuck in the processing queue. This problem is impacting our job execution and overall system performance. Despite previous resolutions, some jobs are once again stalling. Initial investigations are being conducted and we expect to have a resolution shortly. We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. Further updates will be provided regularly.
- identified Dec 05, 2024, 09:09 AM UTC
Summary: We have identified a fault in one of our job dispatcher servers. Upon stopping the problematic server, the jobs that were previously stalled have been freed and will restart according to their schedule. Actions Taken: The faulty job dispatcher server has been stopped to release the stuck jobs. All jobs are now running on a functional server. The problematic server is currently under debugging to determine the root cause of the issue. Next Steps: We will continue monitoring the situation and provide updates as we progress with the debugging process. Thank you for your patience as we work to resolve this issue.
- resolved Dec 05, 2024, 07:43 PM UTC
This incident has been resolved.