SalesRabbit experienced a major incident on June 8, 2023 affecting Web App and Mobile - Android, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 08, 2023, 03:02 PM UTC
We are currently investigating this issue.
- investigating Jun 08, 2023, 03:18 PM UTC
We are continuing to investigate this issue.
- monitoring Jun 08, 2023, 03:47 PM UTC
A network change to AWS caused an issue with our systems being able to communicate with the database. We have implemented a permanent workaround and are seeing systems come back online. We'll continue to monitor the situation.
- resolved Jun 08, 2023, 04:10 PM UTC
This incident has been resolved.
- postmortem Jun 08, 2023, 08:50 PM UTC
We wanted to inform you that at 8:42 AM MT, we received an alert regarding an issue in our system. After a thorough investigation, we discovered that our primary AWS account experienced difficulty in resolving IP addresses for certain databases and cache instances. This problem arose due to a missing configuration that prevented proper communication between our primary environment and the affected services. Upon Identifying this issue, we took immediate action to address this issue with AWS. Our team implemented the necessary changes to establish seamless communication between our primary environment and the affected services. Additionally, we have enabled verified DNS resolution across the connection to ensure smooth functioning when system components are restarted. If you have any further questions or concerns, please don't hesitate to contact our support team.