BombBomb experienced a critical incident on December 7, 2021 affecting Web Application and Chrome Extension and 1 more component, lasting 7h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 07, 2021, 04:16 PM UTC
Amazon web services is down at the moment. Our application is still running, however, you may experience latency.
- identified Dec 07, 2021, 04:25 PM UTC
Amazon has identified the root cause of the issues in the US-EAST-1 Region. This is a network issue in this Region which is impacting multiple services. They're actively working towards recovery.
- identified Dec 07, 2021, 04:44 PM UTC
As the incident with Amazon continues, we are focusing our efforts on how to minimize the impact to us and how to keep our application up throughout this.
- identified Dec 07, 2021, 04:46 PM UTC
We are continuing to work on a fix for this issue.
- identified Dec 07, 2021, 04:48 PM UTC
Our LiveFire notifications are not sending, which is currently not allowing us to collect "Video Watched" events.
- identified Dec 07, 2021, 05:31 PM UTC
The BombBomb application continues to function, however, latency continues.
- identified Dec 07, 2021, 05:49 PM UTC
Our account login page is down due to the Amazon API errors rates. We're focusing our efforts on the login service to see if we can get it working in a different way. It's hosted on AWS Lambda which is unresponsive.
- identified Dec 07, 2021, 06:40 PM UTC
Our log in page continues to be unresponsive. If you've been logged in prior to this page being unresponsive, all parts of the application continue to work with degraded performance with latency issues.
- identified Dec 07, 2021, 07:47 PM UTC
Amazon provided an update on this outage: We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time.
- identified Dec 07, 2021, 08:00 PM UTC
Our engineers continue to work to restore the ability to log in to our application. If you were logged into the web application prior to the log in page going down, the functionality of the BombBomb application remains limited and inconsistent. You may experience issues using our BombBomb extension and recording and saving videos.
- identified Dec 07, 2021, 09:34 PM UTC
Our engineers have restored the ability for you to log in to your BombBomb account. The functionality of the web application has also improved, however, you may still experience latency as you navigate in it.
- identified Dec 07, 2021, 10:20 PM UTC
We received an update from Amazon regarding the AWS outage. They have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. They're continuing to closely monitor the health of the network devices and expect it to continue to make progress towards full recovery. They still do not have an ETA for full recovery at this time.
- identified Dec 07, 2021, 10:49 PM UTC
We are continuing to see latency in the navigation of our application and with the use of our video/screen recorder.
- resolved Dec 07, 2021, 11:30 PM UTC
We will continue to monitor the AWS incident. All of BombBomb's systems are green and operational.