SimpleVoIP experienced a major incident on October 16, 2023 affecting Core Voice Services, lasting 3h 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 16, 2023, 06:39 PM UTC
We have identified a routing issue following our platform upgrade last night that is impacting outbound call delivery to e911 services and causing call completion or one-way/no-way audio issues for certain customers. Our engineers are working to adjust this routing to restore all services as quickly as possible.
- identified Oct 16, 2023, 07:41 PM UTC
In order to resolve the routing issue impacting outbound calls to e911 services, we must restart services which may impact calls in progress. Due to the sensitive nature of the calls being impacted, we are electing to perform these restarts ASAP. The restarts will be performed in a rolling fashion to minimize the impact to your users, but calls lasting longer than 5 minutes will likely be cut off when the restart occurs. We will begin the restarts with our US-East zone at 3:45pm Eastern/12:45pm Pacific and roll through US-Central and US-West shortly after.
- identified Oct 16, 2023, 08:16 PM UTC
Services have been restarted in US-East and traffic has been restored. We have verified that the applied fixes have corrected both the issues with outbound 911 connectivity and the one-way/no-way media issues experienced by certain customers. Engineers are currently working to restart US-Central with minimal impact. Once that is done, we will proceed with US-West.
- monitoring Oct 16, 2023, 09:02 PM UTC
Following the pause of our US-West zone, all calls should now be completing successfully. We will finalize the restart of US-West once all calls in progress have wrapped up, at which point traffic will be restored to primary patterns across all zones. Please reach out to our Support team if your users continue to experience any issues.
- resolved Oct 16, 2023, 09:43 PM UTC
All zones have been restored to full status and are handling traffic normally.