SimpleVoIP experienced a major incident on September 17, 2023 affecting Core Voice Services, lasting 1h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 17, 2023, 06:21 PM UTC
We are investigating reports of calls not completing on our system. We will provide updates as soon as we have identified the cause and impact of this event.
- identified Sep 17, 2023, 06:43 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 17, 2023, 06:50 PM UTC
We have implemented a fix for outbound calling. We are actively investigating potential issues pertaining to some inbound calls failing.
- monitoring Sep 17, 2023, 07:07 PM UTC
Inbound and outbound calls are completing at normal on the platform. Our team will continue to monitor for stability.
- resolved Sep 17, 2023, 07:56 PM UTC
This incident has been resolved.
- postmortem Sep 22, 2023, 06:22 PM UTC
On Sunday, Sept 17th, our monitoring system showed a massive increase in outbound call failures starting at around 12:40pm CDT. Upon investigation, our engineers discovered an issue with one of our redundant carriers used for outbound call delivery. At the time, this carrier was our primary route for outbound calls. This primary route was responding to each call attempt with a failure code that should have resulted in a retry over one of our redundant routes, which would not have been subject to the same issue our experienced by our primary route, and should have resulted in transparent delivery of the call for the SimpleVoIP user. However, our system treated the error codes from our primary route as hard failure codes, terminating the call entirely rather than attempting redelivery over a backup route. Once our engineers identified that automatic failover was not occurring, we manually deprioritized the affected route, moving a redundant carrier route to the primary position, and fully restored outbound service at around 1:40pm CDT. The underlying issue with our primary carrier was addressed, and the manual routing change was reverted Monday morning, Sept 18th. Our engineers are still investigating why we failed to reattempt these calls over our redundant routes. We have confirmed that calls that are undeliverable via our primary carrier are being reattempted via our redundant routes as expected. We have also identified opportunities for monitoring improvements that should reduce our response times and allow us to engage Engineering more directly if similar calling patterns are observed in the future.