Pandium experienced a major incident on September 21, 2024 affecting Admin Dashboard and Runs, lasting 12h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 21, 2024, 01:02 PM UTC
Some runs are firing slowly due to an intermittent issue with token management. We are investigating.
- investigating Sep 21, 2024, 01:02 PM UTC
We are continuing to investigate this issue.
- investigating Sep 21, 2024, 01:35 PM UTC
We have identified the issue and are investigating a fix.
- identified Sep 21, 2024, 02:05 PM UTC
The issue has been identified and a fix has been released
- monitoring Sep 21, 2024, 02:09 PM UTC
The released fix was effective and we are monitoring platform recovery.
- monitoring Sep 21, 2024, 02:37 PM UTC
We are continuing to monitor for any further issues.
- investigating Sep 21, 2024, 03:45 PM UTC
After recovery, we are experiencing a different issue with our underlying platform. We are investigating
- monitoring Sep 21, 2024, 05:06 PM UTC
A fix has been implemented and we are monitoring recovery.
- identified Sep 21, 2024, 06:36 PM UTC
We have identified that there is an issue with our underlying cloud hosting provider and we are working with them to implement a fix. They have escalated this issue and we will provide an update as soon as possible.
- identified Sep 21, 2024, 08:03 PM UTC
We are continuing to work on a fix for this issue.
- identified Sep 21, 2024, 11:35 PM UTC
The Pandium Integration Hub is fully operational however jobs are still running with delays. We will provide an update as soon as possible.
- monitoring Sep 22, 2024, 01:09 AM UTC
A fix has been implemented and runs are recovering. We are closely monitoring and will update when fully resolved.
- resolved Sep 22, 2024, 01:35 AM UTC
The underlying service issue causing instability has been resolved and runs are firing. We are continuing to monitor for the next few hours and will open a new incident if necessary.