brainCloud incident
Another primary mongodb server has become unreachable...
Resolved after 1 hour 40 minutes, affecting Design Portal, on September 27, 2021.
Amazon has identified an issue with EBS instances in one of the zones of us-east-1 where brainCloud is hosted. We have migrated servers out of the affected zone. It turns out that in 2 scenarios, we had multiple instances of the same shard in the same zone - which interfered with MongoDB's ability to fail over when the zone failed. We will put additional processes in place to avoid this configuration issue in the future. Apologies!
Affected components
Update timeline
- investigating Sep 27, 2021, 03:06 AM UTC
We believe there are networking issues happening within the cluster...
- identified Sep 27, 2021, 03:19 AM UTC
It looks like this is an Amazon EBS issue - they are now showing a status that EBS connections in the North Virginia region (where we are) are impaired. This is what is impacting our database.
- resolved Sep 27, 2021, 04:47 AM UTC
Amazon has identified an issue with EBS instances in one of the zones of us-east-1 where brainCloud is hosted. We have migrated servers out of the affected zone. It turns out that in 2 scenarios, we had multiple instances of the same shard in the same zone - which interfered with MongoDB's ability to fail over when the zone failed. We will put additional processes in place to avoid this configuration issue in the future. Apologies!
Your product runs on more than brainCloud
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services