Instabug experienced a major incident on April 18, 2022 affecting Web App, lasting 1h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 18, 2022, 02:41 PM UTC
The Crashes dashboard is not rendering properly for customers. We are aware of the issue and working to promptly resolve it. This does not impact any private-cluster customers.
- identified Apr 18, 2022, 02:43 PM UTC
This issue has been identified and confirmed to be caused by a deployment by our cloud provider. A restore is currently in progress
- monitoring Apr 18, 2022, 02:49 PM UTC
Restore is done. Issue should be now resolved. We're monitoring to make sure everything is working as expected
- resolved Apr 18, 2022, 03:02 PM UTC
This incident has been resolved.
- postmortem May 23, 2022, 04:28 PM UTC
## Summary On April 18th, 2022, between 13:36 UTC and 14:30 UTC, Instabug experienced an operational incident that caused some crashes to not be accessible. During this time, no crashes or occurrences were lost or dropped. This issue only impacted viewing of crashes, and our APIs were not impacted in any way. ## What Happened Amazon Web Services \(AWS\) was performing a blue/green deployment while services underlying Instabug’s infrastructure. During this deployment, several of the indices associated with our Elasticsearch database were incorrectly deleted. Once the issue was discovered, Instabug engineers contacted AWS support who acknowledged the issue and restored the indices from a snapshot. Upon restoration, services were fully functional. ## What We Are Doing About This AWS has committed to notifying us before any release or migration that may impact our services.