Firebolt experienced a major incident on September 28, 2023 affecting AWS ec2-us-east-1, lasting 10h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 28, 2023, 08:10 PM UTC
AWS experience an issue in US-EAST-1 region that may affect the startup of engines. This issue primarily impacts the launch of new instances within this specific region. Our team is actively monitoring the situation, and we are in close communication with AWS to expedite the resolution of this issue. We will provide updates as we receive more information or make progress toward a solution. You can also track it on the Status page of AWS - https://health.aws.amazon.com/health/status
- investigating Sep 28, 2023, 08:24 PM UTC
AWS experience an issue in US-EAST-1 region that may affect the startup of engines. This issue primarily impacts the launch of new instances within this specific region. Our team is actively monitoring the situation, and we are in close communication with AWS to expedite the resolution of this issue. We will provide updates as we receive more information or make progress toward a solution. You can also track it on the Status page of AWS - https://health.aws.amazon.com/health/status
- identified Sep 28, 2023, 08:39 PM UTC
As of now, we are awaiting an update from AWS to resolve the engine startup problem. While we await AWS's response, it's important to note that the issue continues to affect the launch of new instances in the US-EAST-1 region. However, our services in other regions as well as currently running engines remain unaffected and are functioning as expected. Please continue to follow updates here and in AWS status page - https://health.aws.amazon.com/health/status
- monitoring Sep 28, 2023, 09:58 PM UTC
While the AWS outage is not fully resolved, it is gradually recovering, and so do our engines. We continue to monitor our affected engines until we gain a fully operational status. For more updates on the AWS status, please see here: https://health.aws.amazon.com/health/status
- monitoring Sep 28, 2023, 11:24 PM UTC
No significant updates have been provided from AWS. We are monitoring their status page as well as our internal monitoring tools. We do no expect to provide further interim updates, we will update here once the issue is resolved.
- resolved Sep 29, 2023, 06:37 AM UTC
The issue has been resolved and the service is operating normally