Firebolt incident

UI issue - reporting engines states

Major Resolved View vendor source →

Firebolt experienced a major incident on January 19, 2023 affecting Web Application, lasting 3h 15m. The incident has been resolved; the full update timeline is below.

Started
Jan 19, 2023, 03:27 PM UTC
Resolved
Jan 19, 2023, 06:43 PM UTC
Duration
3h 15m
Detected by Pingoru
Jan 19, 2023, 03:27 PM UTC

Affected components

Web Application

Update timeline

  1. investigating Jan 19, 2023, 03:27 PM UTC

    We have a current multi account and multi engine incident. We're still identifying the impact and severity, but as of now it looks like it impacts only the UI and not the engines themselves. We are still checking it.

  2. investigating Jan 19, 2023, 03:44 PM UTC

    We are actively working on a fix for the UI issue. Please do not restart or stop engines until we have the issue resolved. We will have an ETA on a fix in 15 minutes.

  3. investigating Jan 19, 2023, 03:56 PM UTC

    We are still investigating and will update in 10 minutes or when we have a clear ETA

  4. identified Jan 19, 2023, 04:10 PM UTC

    We identified the problem and working on a fix. We will provide an ETA in 15 minutes.

  5. identified Jan 19, 2023, 04:28 PM UTC

    We are still working on a resolution and can confirm in the majority case it is a UI issue. There are a few cases where engines have become unresponsive which we need to fix manually. If an engine is not responsive via the API/SDK, please contact us, and do not restart/stop the engine.

  6. identified Jan 19, 2023, 04:49 PM UTC

    We are currently working through the solution which will propagate across accounts over the next 15 minutes. We will continue to update as we resolve the issue and will mention when it's completely fixed.

  7. identified Jan 19, 2023, 05:15 PM UTC

    We have confirmed the fix and are actively working on applying the fix across all engines. This fix will take approx 1 hour. As mentioned prior this issue should be purely UI related. If you notice connectivity issues during this hour please reach out to support as that is most likely an unrelated issue.

  8. identified Jan 19, 2023, 06:20 PM UTC

    We have resolved the issue with a majority of the engines and are working through the last small fraction of them. We will post when completely resolved. In the interim if you see some engines running and others failed please notify support.

  9. resolved Jan 19, 2023, 06:43 PM UTC

    We have corrected the final set of engines and all services are available. Please contact support if you have any issues.