SimpleVoIP experienced a major incident on August 30, 2020 affecting Core Voice Services, lasting 2h 40m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 30, 2020, 02:19 PM UTC
Level3/CenturyLink, one of the largest internet providers in the world, is having global BGP routing issues. This is causing network routes to flap for us and also transit issues for customer internet traffic coming to/from us in some instances. Our secondary internet links are unaffected. Level3/CenturyLink is aware of the issue and working to resolve as quickly as possible.
- identified Aug 30, 2020, 02:31 PM UTC
CenturyLink is reporting some progress. Here is their latest update: > The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.
- monitoring Aug 30, 2020, 03:00 PM UTC
We are seeing resolution of CenturyLink services in most areas, but many ISPs have made emergency routing changes to attempt to compensate for the Level3 issues that might cause residual impact even after the CenturyLink backbone is fully restored. If any customers continue to experience issues, switching temporarily to a backup internet connection may help.
- resolved Aug 30, 2020, 05:00 PM UTC
CenturyLink has reported that the issues on their end are resolved, and our phone counts show that registrations are back to normal. In case there are lingering issues, the CenturyLink NOC has advised that a reboot of the network equipment at impacted sites may assist in fully restoring services.