Resolved after 27 minutes, affecting Web App and Mobile - Android and 1 more, on April 5, 2023.
We’ve been working on an improved syncing approach, part of onboarding onto the new approach requires a re-sync of leads. Yesterday the feature flag that we use to control who is on the sync was accidentally switched on for all customers. This caused every customer to re-sync their leads all at once which caused our system some stress. We’ve reverted everyone back to the old sync unless they were on the new sync previously. We’ve implemented some training and controls around our feature flags to ensure this is unlikely to happen again.
Affected components
Update timeline
- investigating Apr 05, 2023, 10:57 PM UTC
We are currently investigating a widespread outage.
- monitoring Apr 05, 2023, 11:15 PM UTC
We have implemented a fix. For all affected users, if you continue to have issues you may need to clear local data and log back in.
- resolved Apr 05, 2023, 11:24 PM UTC
This incident has been resolved.
- postmortem Apr 06, 2023, 03:37 PM UTC
We’ve been working on an improved syncing approach, part of onboarding onto the new approach requires a re-sync of leads. Yesterday the feature flag that we use to control who is on the sync was accidentally switched on for all customers. This caused every customer to re-sync their leads all at once which caused our system some stress. We’ve reverted everyone back to the old sync unless they were on the new sync previously. We’ve implemented some training and controls around our feature flags to ensure this is unlikely to happen again.
Your product runs on more than SalesRabbit
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services