Pandium experienced a critical incident on August 1, 2022 affecting Runs, lasting 4h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Runs
Update timeline
- investigating Aug 01, 2022, 12:01 PM UTC
We are currently investigating an issue that is causing some runs to fail.
- investigating Aug 01, 2022, 01:24 PM UTC
We are continuing to investigate this issue.
- investigating Aug 01, 2022, 01:55 PM UTC
We have restored a key service and are continuing to investigate the underlying issue
- identified Aug 01, 2022, 03:09 PM UTC
This issue has been identified and we are working on a fix.
- monitoring Aug 01, 2022, 03:24 PM UTC
A fix has been deployed that has mitigated the issue. There seems to have been a issue where stale configs have been applied. We are continuing to monitor as runs come back online.
- resolved Aug 01, 2022, 04:44 PM UTC
Runs are successfully running. We will continuing to investigating root causes, but immediate issue has been resolved.