Ghost Inspector Outage History

Early this morning, we experienced degraded performance affecting Chrome test execution. Customers may have noticed: - Delayed test runs - Chrome tests were queued longer than usual. - Failed results on our end; during a short window (~14:45 –15:10 UTC), some Chrome runs returned a failed result due to a temporary issue in our system. This is not a problem with your tests. The cause has been resolved, and we've added further system capacity to work through the delay. Test execution is catching up now, and we expect normal performance shortly. Any such failed test runs will re-run normally on their next scheduled execution and can be disregarded. All tests stuck in a queued state should be starting to run over the next 2 hours. Do note that tests using Firefox were not affected. All other Ghost Inspector functionality was also not affected. We greatly apologize for this system disruption. We take system reliability seriously and will do our best to ensure this does not happen again.

This incident has been resolved.

One of our test runner instances began returning intermittent Internal Server Error responses, causing some test runs to fail unexpectedly. The issue was traced to an internal fault on the runner, which we promptly identified and removed from the rotation. We have taken steps to ensure the faulty runner is no longer in use and are continuing to monitor for stability across all test executions. We sincerely apologize for the disruption and appreciate your patience.

Recently, our platform experienced an intermittent error due to one Chrome test runner server among many, which had entered a faulty state and was consistently failing. This issue has now been identified and resolved, and we apologize for any inconvenience this may have caused. To confirm the fix, please restart any tests that previously encountered errors. If you continue to experience any problems, please let us know. Our monitoring systems didn’t detect this sooner because the error rate across all tests stayed within limits. We take this seriously and are working to improve our monitoring to catch similar cases more effectively in the future. Thank you again for your assistance and patience.