SimpleVoIP experienced a major incident on October 4, 2024 affecting Core Voice Services, lasting 1d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 04, 2024, 02:01 PM UTC
We are investigating reports of some calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.
- identified Oct 04, 2024, 02:57 PM UTC
This issue appears so far to be isolated to US-East. Engineers have identified a configuration setting that was removed as part of last night's maintenance. We have restored that setting, and we are restarting services in our US-East region to attempt to resolve the issue.
- identified Oct 04, 2024, 03:01 PM UTC
We are still able to replicate the issue in US-East following the config change and service restart. We are redirecting that traffic over to US-Central as we have been unable to replicate there.
- identified Oct 04, 2024, 04:08 PM UTC
We are still seeing sporadic failures to devices holding on to their US-East registrations. Most devices that have followed the redirect to US-Central appear unaffected. We are identifying the devices that have not moved and pushing them to update their registrations. A power-cycle of the device is usually adequate to clear their DNS cache and move them to the new zone. If you have any locations that are still impacted, please reach out to our support team to provide fresh examples. Our engineers are still looking into the root cause of the issues on EWR as we work to get everyone back to 100%.
- monitoring Oct 04, 2024, 05:10 PM UTC
Our engineers have implemented a permanent fix for the affected server in US-East. There should be no further impact, even for devices that did not follow our redirect to US-Central. We are maintaining the temporary routing away from US-East while we do some additional testing and validation of the fix. We will send a notification before restoring that traffic to its standard route. Please report any impacted calls occurring after 12:30pm EDT to our support team for investigation.
- monitoring Oct 04, 2024, 08:40 PM UTC
We have scheduled a maintenance window for tonight to revert the override away from US-East. We'll keep this incident open and under extra close watch until then.
- resolved Oct 05, 2024, 03:01 PM UTC
Traffic was moved back to US-East overnight, and calls are processing normally across the board. A formal RFO will be provided within a few business days.
- postmortem Oct 16, 2024, 10:37 PM UTC
At approximately 9:30am EDT on Friday, Oct 4, 2024, we started receiving reports of calls dropping after connecting. After gathering examples and collecting more specific descriptions of the symptoms, we were confident that the impact to calls was related to the maintenance performed several hours before, which added a new voice gateway server in each availability zone. To attempt to quickly resolve the issue, we shut off services on those three new servers at around 10:20am EDT, which did not change the rate of failures. All reported failures were from customers homed to our US-East region, so we routed traffic away from US-East at 11am EDT. This fixed the symptoms as soon as phones reregistered to US-Central or US-West, which took less than 5 minutes for most devices. Our Engineering team was able to isolate the root cause of the failed calls to a configuration change made on one of our preexisting voice gateway servers as part of the introduction of the three new servers. This change introduced asymmetric routing between our SIP gateway and our media server, resulting in a perceived call failure once media was established, and causing some strange behavior like rolling a connected call into a user’s voicemail box. The offending config snippet \(originally introduced as a workaround to a problem that no longer exists\) has been removed from our configuration management system so that this issue will not reoccur on future server deployments. We apologize for the impact that this caused to our US-East customers. We have enhanced our post-maintenance QA plan to include more exhaustive server-to-server communication tests so that we can prevent impact from any similar issues in the future.